crawl-sitemap
Crawl to discover all URL locations defined in a sitemap or sitemap index.
Example
import crawl from 'crawl-sitemap';
for await (const url of crawl('https://andrewduthie.com/sitemap.xml')) {
console.log(url);
}
Installation
crawl-sitemap
is authored as an ESM module, and therefore requires Node 12.0 or newer.
Install using NPM or Yarn:
npm install crawl-sitemap
yarn add crawl-sitemap
Usage
The default export will return an async generator when called, which can be iterated asynchronously with for await
to operate on crawled links as soon as they're discovered:
for await (const url of crawl('https://andrewduthie.com/sitemap.xml')) {
console.log(url);
}
API
function crawl(url: string): AsyncGenerator<string>;
License
Copyright 2020 Andrew Duthie
Released under the MIT License. See LICENSE.md.