🚀 Big News: Socket Acquires Coana to Bring Reachability Analysis to Every Appsec Team.Learn more

sitemapper

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

4.0.2
latest
Version published
Weekly downloads
30K
-5.62%
Maintainers
1
Weekly downloads
 
Created

Sitemap-parser

Code Scanning NPM Publish Version Bump Test Codecov CodeFactor GitHub license GitHub release date Inline docs Libraries.io dependency status for latest release license Monthly Downloads npm version release scrutinizer

Parse through a sitemaps xml to get all the urls for your crawler.

Installation

npm install sitemapper --save

Simple Example

const Sitemapper = require('sitemapper');

const sitemap = new Sitemapper();

sitemap.fetch('https://wp.seantburke.com/sitemap.xml').then(function (sites) {
  console.log(sites);
});

Examples

import Sitemapper from 'sitemapper';

(async () => {
  const Google = new Sitemapper({
    url: 'https://www.google.com/work/sitemap.xml',
    timeout: 15000, // 15 seconds
  });

  try {
    const { sites } = await Google.fetch();
    console.log(sites);
  } catch (error) {
    console.log(error);
  }
})();

// or

const sitemapper = new Sitemapper();
sitemapper.timeout = 5000;

sitemapper
  .fetch('https://wp.seantburke.com/sitemap.xml')
  .then(({ url, sites }) => console.log(`url:${url}`, 'sites:', sites))
  .catch((error) => console.log(error));

Options

You can add options on the initial Sitemapper object when instantiating it.

  • requestHeaders: (Object) - Additional Request Headers (e.g. User-Agent)
  • timeout: (Number) - Maximum timeout in ms for a single URL. Default: 15000 (15 seconds)
  • url: (String) - Sitemap URL to crawl
  • debug: (Boolean) - Enables/Disables debug console logging. Default: False
  • concurrency: (Number) - Sets the maximum number of concurrent sitemap crawling threads. Default: 10
  • retries: (Number) - Sets the maximum number of retries to attempt in case of an error response (e.g. 404 or Timeout). Default: 0
  • rejectUnauthorized: (Boolean) - If true, it will throw on invalid certificates, such as expired or self-signed ones. Default: True
  • lastmod: (Number) - Timestamp of the minimum lastmod value allowed for returned urls
  • proxyAgent: (HttpProxyAgent|HttpsProxyAgent) - instance of npm "hpagent" HttpProxyAgent or HttpsProxyAgent to be passed to npm "got"
  • exclusions: (Array) - Array of regex patterns to exclude URLs from being processed
  • fields: (Object) - An object of fields to be returned from the sitemap. Leaving a field out has the same effect as <field>: false. If not specified sitemapper defaults to returning the 'classic' array of urls. Available fields:
    • loc: (Boolean) - The URL location of the page
    • sitemap: (Boolean) - The URL of the sitemap containing the URL, useful if was used in the sitemap
    • lastmod: (Boolean) - The date of last modification of the page
    • changefreq: (Boolean) - How frequently the page is likely to change
    • priority: (Boolean) - The priority of this URL relative to other URLs on your site
    • image:loc: (Boolean) - The URL location of the image (for image sitemaps)
    • image:title: (Boolean) - The title of the image (for image sitemaps)
    • image:caption: (Boolean) - The caption of the image (for image sitemaps)
    • video:title: (Boolean) - The title of the video (for video sitemaps)
    • video:description: (Boolean) - The description of the video (for video sitemaps)
    • video:thumbnail_loc: (Boolean) - The thumbnail URL of the video (for video sitemaps)

For Example:

fields: {
  loc: true,
  lastmod: true,
  changefreq: true,
  priority: true,
}

Leaving a field out has the same effect as <field>: false. If not specified sitemapper defaults to returning the 'classic' array of urls.

An example using all available options:

import { HttpsProxyAgent } from 'hpagent';

const sitemapper = new Sitemapper({
  requestHeaders: {
    'User-Agent':
      'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
  },
  timeout: 15000,
  url: 'https://art-works.community/sitemap.xml',
  debug: true,
  concurrency: 2,
  retries: 1,
  lastmod: 1600000000000,
  proxyAgent: new HttpsProxyAgent({
    proxy: 'http://localhost:8080',
  }),
  exclusions: [/\/v1\//, /scary/],
  rejectUnauthorized: false,
  fields: {
    loc: true,
    lastmod: true,
    priority: true,
    changefreq: true,
    sitemap: true,
  },
});

FAQs

Package last updated on 16 May 2025

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts