Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
Socket
Sign inDemoInstall
Socket
Sign inDemoInstall

sitemap

Package Overview
Dependencies
Maintainers
1
Versions
73
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

sitemap - npm Package Compare versions

Comparing version 4.1.1 to 5.0.0

dist/lib/sitemap-stream.d.ts
dist/lib/sitemap-stream.js

25

CHANGELOG.md

@@ -1,4 +0,25 @@

# master
Nothing yet
# 5.0.0
## Streams
This release is heavily focused on converting the core methods of this library to use streams. Why? Overall its made the API ~20% faster and uses only 10% or less of the memory. Some tradeoffs had to be made as in their nature streams are operate on individual segments of data as opposed to the whole. For instance, the streaming interface does not support removal of sitemap items as it does not hold on to a sitemap item after its converted to XML. It should however be possible to create your own transform that filters out entries should you desire it. The existing synchronous interfaces will remain for this release at least. Do not be surprised if they go away in a future breaking release.
## Sitemap Index
This library interface has been overhauled to use streams internally. Although it would have been preferable to convert this to a stream as well, I could not think of an interface that wouldn't actually end up more complex or confusing. It may be altered in the near future to accept a stream in addition to a simple list.
## Misc
- runnable examples, some pulled straight from README have been added to the examples directory.
- createSitemapsIndex was renamed createSitemapsAndIndex to more accurately reflect its function. It now returns a promise that resolves to true or throws with an error.
- You can now add to existing sitemap.xml files via the cli using `npx sitemap --prepend existingSitemap.xml < listOfNewURLs.json.txt`
## Breaking Changes
- Dropped support for mobile sitemap - Google appears to have deleted their dtd and all references to it, strongly implying that they do not want you to use it. As its absence now breaks the validator, it has been dropped.
- normalizeURL(url, XMLRoot, hostname) -> normalizeURL(url, hostname)
- The second argument was unused and has been eliminated
- Support for Node 8 dropped - Node 8 is reaching its EOL December 2019
- xslURL is being dropped from all apis - styling xml is out of scope of this library.
- createSitemapIndex has been converted to a promised based api rather than callback.
- createSitemapIndex now gzips by default - pass gzip: false to disable
- cacheTime is being dropped from createSitemapIndex - This didn't actually cache the way it was written so this should be a non-breaking change in effect.
- SitemapIndex as a class has been dropped. The class did all its work on construction and there was no reason to hold on to it once you created it.
- The options for the cli have been overhauled
- `--json` is now inferred
- `--line-separated` has been flipped to `--single-line-json` to by default output options immediately compatible with feeding back into sitemap
# 4.1.1

@@ -5,0 +26,0 @@ Add a pretty print option to `toString(false)`

75

dist/cli.js
#!/usr/bin/env node
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : new P(function (resolve) { resolve(result.value); }).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
Object.defineProperty(exports, "__esModule", { value: true });
const index_1 = require("./index");
const readline_1 = require("readline");
const fs_1 = require("fs");

@@ -18,39 +8,13 @@ const xmllint_1 = require("./lib/xmllint");

const sitemap_parser_1 = require("./lib/sitemap-parser");
console.warn('CLI is new and likely to change quite a bit. Please send feature/bug requests to https://github.com/ekalinin/sitemap.js/issues');
const utils_1 = require("./lib/utils");
const sitemap_stream_1 = require("./lib/sitemap-stream");
/* eslint-disable-next-line @typescript-eslint/no-var-requires */
const arg = require('arg');
const preamble = '<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">';
const closetag = '</urlset>';
let first = true;
const println = (line) => {
if (first) {
first = false;
process.stdout.write(preamble);
}
process.stdout.write(index_1.SitemapItem.justItem(index_1.Sitemap.normalizeURL(line)));
};
function processStreams(streams, isJSON = false) {
return __awaiter(this, void 0, void 0, function* () {
for (let stream of streams) {
yield new Promise((resolve) => {
const rl = readline_1.createInterface({
input: stream
});
rl.on('line', (line) => println(isJSON ? JSON.parse(line) : line));
rl.on('close', () => {
resolve();
});
});
}
process.stdout.write(closetag);
return true;
});
}
const argSpec = {
'--help': Boolean,
'--version': Boolean,
'--json': Boolean,
'--validate': Boolean,
'--parse': Boolean,
'--line-separated': Boolean
'--single-line-json': Boolean,
'--prepend': String
};

@@ -80,20 +44,13 @@ const argv = arg(argSpec);

--version Print the version
--json Parse each line as json and feed to Sitemap
--parse Parse fed xml and spit out config
--line-separated When used with parse, it spits out each entry as json rather
than the whole json. This can be then consumed with --json by
the cli
--prepend sitemap.xml < urlsToAdd.json
--single-line-json When used with parse, it spits out each entry as json rather
than the whole json.
`);
}
else if (argv['--parse']) {
sitemap_parser_1.parseSitemap(getStream()).then((items) => {
if (argv['--line-separated'] && items.urls) {
items.urls.forEach((url) => {
console.log(JSON.stringify(url));
});
}
else {
console.log(JSON.stringify(items));
}
});
getStream()
.pipe(new sitemap_parser_1.XMLToISitemapOptions())
.pipe(new sitemap_parser_1.ObjectStreamToJSON({ lineSeparated: !argv["--single-line-json"] }))
.pipe(process.stdout);
}

@@ -121,4 +78,12 @@ else if (argv['--validate']) {

}
processStreams(streams, argv['--json']);
const sms = new sitemap_stream_1.SitemapStream();
if (argv['--prepend']) {
fs_1.createReadStream(argv["--prepend"])
.pipe(new sitemap_parser_1.XMLToISitemapOptions())
.pipe(sms);
}
utils_1.lineSeparatedURLsToSitemapOptions(utils_1.mergeStreams(streams))
.pipe(sms)
.pipe(process.stdout);
}
//# sourceMappingURL=cli.js.map

4

dist/index.d.ts

@@ -10,6 +10,8 @@ /*!

export * from './lib/sitemap-index';
export * from './lib/sitemap-stream';
export * from './lib/errors';
export * from './lib/types';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, validateSMIOptions } from './lib/utils';
export { xmlLint } from './lib/xmllint';
export { parseSitemap } from './lib/sitemap-parser';
export { parseSitemap, XMLToISitemapOptions, ObjectStreamToJSON } from './lib/sitemap-parser';
export default createSitemap;

7

dist/index.js

@@ -15,4 +15,9 @@ "use strict";

__export(require("./lib/sitemap-index"));
__export(require("./lib/sitemap-stream"));
__export(require("./lib/errors"));
__export(require("./lib/types"));
var utils_1 = require("./lib/utils");
exports.lineSeparatedURLsToSitemapOptions = utils_1.lineSeparatedURLsToSitemapOptions;
exports.mergeStreams = utils_1.mergeStreams;
exports.validateSMIOptions = utils_1.validateSMIOptions;
var xmllint_1 = require("./lib/xmllint");

@@ -22,3 +27,5 @@ exports.xmlLint = xmllint_1.xmlLint;

exports.parseSitemap = sitemap_parser_1.parseSitemap;
exports.XMLToISitemapOptions = sitemap_parser_1.XMLToISitemapOptions;
exports.ObjectStreamToJSON = sitemap_parser_1.ObjectStreamToJSON;
exports.default = sitemap_1.createSitemap;
//# sourceMappingURL=index.js.map

3

dist/lib/errors.d.ts

@@ -45,2 +45,5 @@ /*!

}
export declare class InvalidVideoRating extends Error {
constructor(message?: string);
}
export declare class InvalidAttrValue extends Error {

@@ -47,0 +50,0 @@ constructor(key: string, val: any, validator: RegExp);

21

dist/lib/errors.js

@@ -15,3 +15,2 @@ "use strict";

this.name = 'NoURLError';
// @ts-ignore
Error.captureStackTrace(this, NoURLError);

@@ -28,3 +27,2 @@ }

this.name = 'NoConfigError';
// @ts-ignore
Error.captureStackTrace(this, NoConfigError);

@@ -41,3 +39,2 @@ }

this.name = 'ChangeFreqInvalidError';
// @ts-ignore
Error.captureStackTrace(this, ChangeFreqInvalidError);

@@ -54,3 +51,2 @@ }

this.name = 'PriorityInvalidError';
// @ts-ignore
Error.captureStackTrace(this, PriorityInvalidError);

@@ -67,3 +63,2 @@ }

this.name = 'UndefinedTargetFolder';
// @ts-ignore
Error.captureStackTrace(this, UndefinedTargetFolder);

@@ -77,3 +72,2 @@ }

this.name = 'InvalidVideoFormat';
// @ts-ignore
Error.captureStackTrace(this, InvalidVideoFormat);

@@ -87,3 +81,2 @@ }

this.name = 'InvalidVideoDuration';
// @ts-ignore
Error.captureStackTrace(this, InvalidVideoDuration);

@@ -97,3 +90,2 @@ }

this.name = 'InvalidVideoDescription';
// @ts-ignore
Error.captureStackTrace(this, InvalidVideoDescription);

@@ -103,2 +95,10 @@ }

exports.InvalidVideoDescription = InvalidVideoDescription;
class InvalidVideoRating extends Error {
constructor(message) {
super(message || 'rating must be between 0 and 5');
this.name = 'InvalidVideoRating';
Error.captureStackTrace(this, InvalidVideoRating);
}
}
exports.InvalidVideoRating = InvalidVideoRating;
class InvalidAttrValue extends Error {

@@ -109,3 +109,2 @@ // eslint-disable-next-line @typescript-eslint/no-explicit-any

this.name = 'InvalidAttrValue';
// @ts-ignore
Error.captureStackTrace(this, InvalidAttrValue);

@@ -121,3 +120,2 @@ }

this.name = 'InvalidAttr';
// @ts-ignore
Error.captureStackTrace(this, InvalidAttr);

@@ -131,3 +129,2 @@ }

this.name = 'InvalidNewsFormat';
// @ts-ignore
Error.captureStackTrace(this, InvalidNewsFormat);

@@ -141,3 +138,2 @@ }

this.name = 'InvalidNewsAccessValue';
// @ts-ignore
Error.captureStackTrace(this, InvalidNewsAccessValue);

@@ -151,3 +147,2 @@ }

this.name = 'XMLLintUnavailable';
// @ts-ignore
Error.captureStackTrace(this, XMLLintUnavailable);

@@ -154,0 +149,0 @@ }

72

dist/lib/sitemap-index.d.ts

@@ -1,28 +0,3 @@

import { ICallback, ISitemapIndexItemOptions, SitemapItemOptions } from './types';
import { ISitemapIndexItemOptions, ISitemapItemOptionsLoose } from './types';
/**
* Shortcut for `new SitemapIndex (...)`.
* Create several sitemaps and an index automatically from a list of urls
*
* @param {Object} conf
* @param {String|Array} conf.urls
* @param {String} conf.targetFolder
* @param {String} conf.hostname
* @param {Number} conf.cacheTime
* @param {String} conf.sitemapName
* @param {Number} conf.sitemapSize
* @param {String} conf.xslUrl
* @return {SitemapIndex}
*/
export declare function createSitemapIndex(conf: {
urls: SitemapIndex["urls"];
targetFolder: SitemapIndex["targetFolder"];
hostname?: SitemapIndex["hostname"];
cacheTime?: SitemapIndex["cacheTime"];
sitemapName?: SitemapIndex["sitemapName"];
sitemapSize?: SitemapIndex["sitemapSize"];
xslUrl?: SitemapIndex["xslUrl"];
gzip?: boolean;
callback?: SitemapIndex["callback"];
}): SitemapIndex;
/**
* Builds a sitemap index from urls

@@ -39,3 +14,2 @@ *

urls: (ISitemapIndexItemOptions | string)[];
xslUrl?: string;
xmlNs?: string;

@@ -45,29 +19,21 @@ lastmod?: number | string;

/**
* Sitemap index (for several sitemaps)
* Shortcut for `new SitemapIndex (...)`.
* Create several sitemaps and an index automatically from a list of urls
*
* @param {Object} conf
* @param {String|Array} conf.urls
* @param {String} conf.targetFolder where do you want the generated index and maps put
* @param {String} conf.hostname required for index file, will also be used as base url for sitemap items
* @param {String} conf.sitemapName what do you want to name the files it generats
* @param {Number} conf.sitemapSize maximum number of entries a sitemap should have before being split
* @param {Boolean} conf.gzip whether to gzip the files (defaults to true)
* @return {SitemapIndex}
*/
declare class SitemapIndex {
urls: (string | SitemapItemOptions)[];
export declare function createSitemapsAndIndex({ urls, targetFolder, hostname, sitemapName, sitemapSize, gzip, }: {
urls: (string | ISitemapItemOptionsLoose)[];
targetFolder: string;
hostname?: string | undefined;
sitemapSize?: number | undefined;
xslUrl?: string | undefined;
callback?: ICallback<Error, boolean> | undefined;
sitemapName: string;
sitemapId: number;
sitemaps: string[];
chunks: (string | SitemapItemOptions)[][];
cacheTime?: number;
/**
* @param {String|Array} urls
* @param {String} targetFolder
* @param {String} hostname optional
* @param {Number} cacheTime optional in milliseconds
* @param {String} sitemapName optional
* @param {Number} sitemapSize optional This limit is defined by Google. See: https://sitemaps.org/protocol.php#index
* @param {Number} xslUrl optional
* @param {Boolean} gzip optional
* @param {Function} callback optional
*/
constructor(urls?: (string | SitemapItemOptions)[], targetFolder?: string, hostname?: string | undefined, cacheTime?: number, sitemapName?: string, sitemapSize?: number | undefined, xslUrl?: string | undefined, gzip?: boolean, callback?: ICallback<Error, boolean> | undefined);
}
export {};
hostname?: string;
sitemapName?: string;
sitemapSize?: number;
gzip?: boolean;
}): Promise<boolean>;

151

dist/lib/sitemap-index.js
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
Object.defineProperty(exports, "__esModule", { value: true });
const util_1 = require("util");
const fs_1 = require("fs");
const xmlbuilder_1 = require("xmlbuilder");
const sitemap_1 = require("./sitemap");
const errors_1 = require("./errors");
const utils_1 = require("./utils");
const sitemap_stream_1 = require("./sitemap-stream");
const zlib_1 = require("zlib");
const statPromise = util_1.promisify(fs_1.stat);
/**
* Shortcut for `new SitemapIndex (...)`.
* Create several sitemaps and an index automatically from a list of urls
*
* @param {Object} conf
* @param {String|Array} conf.urls
* @param {String} conf.targetFolder
* @param {String} conf.hostname
* @param {Number} conf.cacheTime
* @param {String} conf.sitemapName
* @param {Number} conf.sitemapSize
* @param {String} conf.xslUrl
* @return {SitemapIndex}
*/
function createSitemapIndex(conf) {
// cleaner diff
// eslint-disable-next-line @typescript-eslint/no-use-before-define
return new SitemapIndex(conf.urls, conf.targetFolder, conf.hostname, conf.cacheTime, conf.sitemapName, conf.sitemapSize, conf.xslUrl, conf.gzip, conf.callback);
}
exports.createSitemapIndex = createSitemapIndex;
/**
* Builds a sitemap index from urls

@@ -41,5 +33,2 @@ *

let lastmod = '';
if (conf.xslUrl) {
root.instructionBefore('xml-stylesheet', `type="text/xsl" href="${conf.xslUrl}"`);
}
if (!conf.xmlNs) {

@@ -49,3 +38,3 @@ conf.xmlNs = 'xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';

const ns = conf.xmlNs.split(' ');
for (let attr of ns) {
for (const attr of ns) {
const [k, v] = attr.split('=');

@@ -75,33 +64,21 @@ root.attribute(k, v.replace(/^['"]|['"]$/g, ''));

/**
* Sitemap index (for several sitemaps)
* Shortcut for `new SitemapIndex (...)`.
* Create several sitemaps and an index automatically from a list of urls
*
* @param {Object} conf
* @param {String|Array} conf.urls
* @param {String} conf.targetFolder where do you want the generated index and maps put
* @param {String} conf.hostname required for index file, will also be used as base url for sitemap items
* @param {String} conf.sitemapName what do you want to name the files it generats
* @param {Number} conf.sitemapSize maximum number of entries a sitemap should have before being split
* @param {Boolean} conf.gzip whether to gzip the files (defaults to true)
* @return {SitemapIndex}
*/
class SitemapIndex {
/**
* @param {String|Array} urls
* @param {String} targetFolder
* @param {String} hostname optional
* @param {Number} cacheTime optional in milliseconds
* @param {String} sitemapName optional
* @param {Number} sitemapSize optional This limit is defined by Google. See: https://sitemaps.org/protocol.php#index
* @param {Number} xslUrl optional
* @param {Boolean} gzip optional
* @param {Function} callback optional
*/
constructor(urls = [], targetFolder = '.', hostname, cacheTime, sitemapName, sitemapSize, xslUrl, gzip = false, callback) {
this.urls = urls;
this.targetFolder = targetFolder;
this.hostname = hostname;
this.sitemapSize = sitemapSize;
this.xslUrl = xslUrl;
this.callback = callback;
if (sitemapName === undefined) {
this.sitemapName = 'sitemap';
}
else {
this.sitemapName = sitemapName;
}
this.sitemapId = 0;
this.sitemaps = [];
function createSitemapsAndIndex({ urls, targetFolder, hostname, sitemapName = 'sitemap', sitemapSize = 50000, gzip = true, }) {
return __awaiter(this, void 0, void 0, function* () {
let sitemapId = 0;
const sitemapPaths = [];
try {
if (!fs_1.statSync(targetFolder).isDirectory()) {
const stats = yield statPromise(targetFolder);
if (!stats.isDirectory()) {
throw new errors_1.UndefinedTargetFolder();

@@ -113,39 +90,41 @@ }

}
this.chunks = utils_1.chunk(urls, this.sitemapSize);
let processesCount = this.chunks.length + 1;
this.chunks.forEach((chunk, index) => {
const extension = '.xml' + (gzip ? '.gz' : '');
const filename = this.sitemapName + '-' + this.sitemapId++ + extension;
this.sitemaps.push(filename);
let sitemap = sitemap_1.createSitemap({
hostname,
cacheTime,
urls: chunk,
xslUrl
});
let stream = fs_1.createWriteStream(targetFolder + '/' + filename);
stream.once('open', (fd) => {
stream.write(gzip ? sitemap.toGzip() : sitemap.toString());
stream.end();
processesCount--;
if (processesCount === 0 && typeof this.callback === 'function') {
this.callback(undefined, true);
const chunks = utils_1.chunk(urls, sitemapSize);
const smPromises = chunks.map((chunk) => {
return new Promise((resolve, reject) => {
const extension = '.xml' + (gzip ? '.gz' : '');
const filename = sitemapName + '-' + sitemapId++ + extension;
sitemapPaths.push(filename);
const ws = fs_1.createWriteStream(targetFolder + '/' + filename);
const sms = new sitemap_stream_1.SitemapStream({ hostname });
let pipe;
if (gzip) {
pipe = sms.pipe(zlib_1.createGzip()).pipe(ws);
}
else {
pipe = sms.pipe(ws);
}
chunk.forEach(smi => sms.write(smi));
sms.end();
pipe.on('finish', () => resolve(true));
pipe.on('error', (e) => reject(e));
});
});
const stream = fs_1.createWriteStream(targetFolder + '/' +
this.sitemapName + '-index.xml');
stream.once('open', (fd) => {
stream.write(buildSitemapIndex({
urls: this.sitemaps.map((sitemap) => hostname + '/' + sitemap),
xslUrl
}));
stream.end();
processesCount--;
if (processesCount === 0 && typeof this.callback === 'function') {
this.callback(undefined, true);
}
const indexPromise = new Promise((resolve, reject) => {
const indexWS = fs_1.createWriteStream(targetFolder + "/" + sitemapName + "-index.xml");
indexWS.once('open', (fd) => {
indexWS.write(buildSitemapIndex({
urls: sitemapPaths.map((smPath) => hostname + '/' + smPath)
}));
indexWS.end();
});
indexWS.on('finish', () => resolve(true));
indexWS.on('error', (e) => reject(e));
});
}
return Promise.all([
indexPromise,
...smPromises
]).then(() => true);
});
}
exports.createSitemapsAndIndex = createSitemapsAndIndex;
//# sourceMappingURL=sitemap-index.js.map

1

dist/lib/sitemap-item.d.ts

@@ -18,3 +18,2 @@ import { XMLElement } from 'xmlbuilder';

androidLink?: SitemapItemOptions["androidLink"];
mobile?: SitemapItemOptions["mobile"];
video?: SitemapItemOptions["video"];

@@ -21,0 +20,0 @@ ampLink?: SitemapItemOptions["ampLink"];

86

dist/lib/sitemap-item.js

@@ -15,3 +15,3 @@ "use strict";

if (conf[key] !== undefined) {
let keyAr = key.split(':');
const keyAr = key.split(':');
if (keyAr.length !== 2) {

@@ -49,3 +49,2 @@ throw new errors_1.InvalidAttr(key);

this.androidLink = conf.androidLink;
this.mobile = conf.mobile;
this.video = conf.video;

@@ -79,10 +78,10 @@ this.ampLink = conf.ampLink;

const videoxml = this.url.element('video:video');
videoxml.element('video:thumbnail_loc', video.thumbnail_loc);
videoxml.element('video:title').cdata(video.title);
videoxml.element('video:description').cdata(video.description);
videoxml.element('video:thumbnail_loc').text(video.thumbnail_loc);
videoxml.element('video:title').text(video.title);
videoxml.element('video:description').text(video.description);
if (video.content_loc) {
videoxml.element('video:content_loc', video.content_loc);
videoxml.element('video:content_loc').text(video.content_loc);
}
if (video.player_loc) {
videoxml.element('video:player_loc', attrBuilder(video, 'player_loc:autoplay'), video.player_loc);
videoxml.element('video:player_loc', attrBuilder(video, 'player_loc:autoplay')).text(video.player_loc);
}

@@ -93,3 +92,3 @@ if (video.duration) {

if (video.expiration_date) {
videoxml.element('video:expiration_date', video.expiration_date);
videoxml.element('video:expiration_date').text(video.expiration_date);
}

@@ -103,36 +102,36 @@ if (video.rating !== undefined) {

if (video.publication_date) {
videoxml.element('video:publication_date', video.publication_date);
videoxml.element('video:publication_date').text(video.publication_date);
}
for (const tag of video.tag) {
videoxml.element('video:tag', tag);
videoxml.element('video:tag').text(tag);
}
if (video.category) {
videoxml.element('video:category', video.category);
videoxml.element('video:category').text(video.category);
}
if (video.family_friendly) {
videoxml.element('video:family_friendly', video.family_friendly);
videoxml.element('video:family_friendly').text(video.family_friendly);
}
if (video.restriction) {
videoxml.element('video:restriction', attrBuilder(video, 'restriction:relationship'), video.restriction);
videoxml.element('video:restriction', attrBuilder(video, 'restriction:relationship')).text(video.restriction);
}
if (video.gallery_loc) {
videoxml.element('video:gallery_loc', { title: video['gallery_loc:title'] }, video.gallery_loc);
videoxml.element('video:gallery_loc', { title: video['gallery_loc:title'] }).text(video.gallery_loc);
}
if (video.price) {
videoxml.element('video:price', attrBuilder(video, ['price:resolution', 'price:currency', 'price:type']), video.price);
videoxml.element('video:price', attrBuilder(video, ['price:resolution', 'price:currency', 'price:type'])).text(video.price);
}
if (video.requires_subscription) {
videoxml.element('video:requires_subscription', video.requires_subscription);
videoxml.element('video:requires_subscription').text(video.requires_subscription);
}
if (video.uploader) {
videoxml.element('video:uploader', video.uploader);
videoxml.element('video:uploader').text(video.uploader);
}
if (video.platform) {
videoxml.element('video:platform', attrBuilder(video, 'platform:relationship'), video.platform);
videoxml.element('video:platform', attrBuilder(video, 'platform:relationship')).text(video.platform);
}
if (video.live) {
videoxml.element('video:live', video.live);
videoxml.element('video:live').text(video.live);
}
if (video.id) {
videoxml.element('video:id', { type: 'url' }, video.id);
videoxml.element('video:id', { type: 'url' }).text(video.id);
}

@@ -146,2 +145,3 @@ }

this.url.children = [];
// eslint-disable-next-line @typescript-eslint/ban-ts-ignore
// @ts-ignore

@@ -162,14 +162,14 @@ this.url.attribs = {};

const xmlObj = {};
xmlObj['image:loc'] = image.url;
xmlObj['image:loc'] = { '#text': image.url };
if (image.caption) {
xmlObj['image:caption'] = { '#cdata': image.caption };
xmlObj['image:caption'] = { '#text': image.caption };
}
if (image.geoLocation) {
xmlObj['image:geo_location'] = image.geoLocation;
xmlObj['image:geo_location'] = { '#text': image.geoLocation };
}
if (image.title) {
xmlObj['image:title'] = { '#cdata': image.title };
xmlObj['image:title'] = { '#text': image.title };
}
if (image.license) {
xmlObj['image:license'] = image.license;
xmlObj['image:license'] = { '#text': image.license };
}

@@ -192,3 +192,3 @@ this.url.element({ 'image:image': xmlObj });

else if (this.expires && p === 'expires') {
this.url.element('expires', new Date(this.expires).toISOString());
this.url.element('expires').text(new Date(this.expires).toISOString());
}

@@ -198,11 +198,5 @@ else if (this.androidLink && p === 'androidLink') {

}
else if (this.mobile && p === 'mobile') {
const mobileitem = this.url.element('mobile:mobile');
if (typeof this.mobile === 'string') {
mobileitem.att('type', this.mobile);
}
}
else if (this.priority !== undefined && p === 'priority') {
if (this.conf.fullPrecisionPriority) {
this.url.element(p, this.priority + '');
this.url.element(p).text(this.priority + '');
}

@@ -217,25 +211,25 @@ else {

else if (this.news && p === 'news') {
let newsitem = this.url.element('news:news');
const newsitem = this.url.element('news:news');
if (this.news.publication) {
let publication = newsitem.element('news:publication');
const publication = newsitem.element('news:publication');
if (this.news.publication.name) {
publication.element('news:name').cdata(this.news.publication.name);
publication.element('news:name').text(this.news.publication.name);
}
if (this.news.publication.language) {
publication.element('news:language', this.news.publication.language);
publication.element('news:language').text(this.news.publication.language);
}
}
if (this.news.access) {
newsitem.element('news:access', this.news.access);
newsitem.element('news:access').text(this.news.access);
}
if (this.news.genres) {
newsitem.element('news:genres', this.news.genres);
newsitem.element('news:genres').text(this.news.genres);
}
newsitem.element('news:publication_date', this.news.publication_date);
newsitem.element('news:title').cdata(this.news.title);
newsitem.element('news:publication_date').text(this.news.publication_date);
newsitem.element('news:title').text(this.news.title);
if (this.news.keywords) {
newsitem.element('news:keywords', this.news.keywords);
newsitem.element('news:keywords').text(this.news.keywords);
}
if (this.news.stock_tickers) {
newsitem.element('news:stock_tickers', this.news.stock_tickers);
newsitem.element('news:stock_tickers').text(this.news.stock_tickers);
}

@@ -251,9 +245,9 @@ }

else if (this.loc && p === 'loc') {
this.url.element(p, this.loc);
this.url.element(p).text(this.loc);
}
else if (this.changefreq && p === 'changefreq') {
this.url.element(p, this.changefreq);
this.url.element(p).text(this.changefreq);
}
else if (this.lastmod && p === 'lastmod') {
this.url.element(p, this.lastmod);
this.url.element(p).text(this.lastmod);
}

@@ -260,0 +254,0 @@ }

31

dist/lib/sitemap-parser.d.ts
/// <reference types="node" />
import { Readable } from 'stream';
import { SAXStream } from 'sax';
import { Readable, Transform, TransformOptions, TransformCallback } from 'stream';
import { SitemapItemOptions, ErrorLevel } from "./types";
import { ISitemapOptions } from './sitemap';
export interface ISitemapStreamParseOpts extends TransformOptions, Pick<ISitemapOptions, 'level'> {
}
/**
* Takes a stream of xml and transforms it into a stream of ISitemapOptions
* Use this to parse existing sitemaps into config options compatible with this library
*/
export declare class XMLToISitemapOptions extends Transform {
level: ErrorLevel;
saxStream: SAXStream;
constructor(opts?: ISitemapStreamParseOpts);
_transform(data: string, encoding: string, callback: TransformCallback): void;
}
/**
Read xml and resolve with the configuration that would produce it or reject with

@@ -22,1 +36,16 @@ an error

export declare function parseSitemap(xml: Readable): Promise<ISitemapOptions>;
export interface IObjectToStreamOpts extends TransformOptions {
lineSeparated: boolean;
}
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
export declare class ObjectStreamToJSON extends Transform {
lineSeparated: boolean;
firstWritten: boolean;
constructor(opts?: IObjectToStreamOpts);
_transform(chunk: SitemapItemOptions, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}

270

dist/lib/sitemap-parser.js
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : new P(function (resolve) { resolve(result.value); }).then(fulfilled, rejected); }
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());

@@ -15,15 +16,21 @@ });

const sax_1 = __importDefault(require("sax"));
const tagTemplate = {
img: [],
video: [],
links: [],
url: ''
};
const videoTemplate = {
tag: [],
// eslint-disable-next-line @typescript-eslint/camelcase
thumbnail_loc: "",
title: "",
description: ""
};
const stream_1 = require("stream");
const types_1 = require("./types");
function tagTemplate() {
return {
img: [],
video: [],
links: [],
url: ''
};
}
function videoTemplate() {
return {
tag: [],
// eslint-disable-next-line @typescript-eslint/camelcase
thumbnail_loc: "",
title: "",
description: ""
};
}
const imageTemplate = {

@@ -36,42 +43,40 @@ url: ''

};
function newsTemplate() {
return {
publication: { name: "", language: "" },
// eslint-disable-next-line @typescript-eslint/camelcase
publication_date: "",
title: ""
};
}
const defaultStreamOpts = {};
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemap, createSitemap } = require('sitemap')
parseSitemap(createReadStream('./example.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@return {Promise<ISitemapOptions>} resolves with a valid config that can be
passed to createSitemap. Rejects with an Error object.
* Takes a stream of xml and transforms it into a stream of ISitemapOptions
* Use this to parse existing sitemaps into config options compatible with this library
*/
function parseSitemap(xml) {
return __awaiter(this, void 0, void 0, function* () {
// @ts-ignore
const saxStream = sax_1.default.createStream(true, { xmlns: true, strictEntities: true, trim: true });
const smi = [];
let currentItem = Object.assign({}, tagTemplate);
class XMLToISitemapOptions extends stream_1.Transform {
constructor(opts = defaultStreamOpts) {
opts.objectMode = true;
super(opts);
this.saxStream = sax_1.default.createStream(true, {
xmlns: true,
// eslint-disable-next-line @typescript-eslint/ban-ts-ignore
// @ts-ignore
strictEntities: true,
trim: true
});
this.level = opts.level || types_1.ErrorLevel.WARN;
let currentItem = tagTemplate();
let currentTag;
let currentVideo = Object.assign({}, videoTemplate);
let currentVideo = videoTemplate();
let currentImage = Object.assign({}, imageTemplate);
let currentLink = Object.assign({}, linkTemplate);
let dontpushCurrentLink = false;
saxStream.on('opentagstart', (tag) => {
this.saxStream.on('opentagstart', (tag) => {
currentTag = tag.name;
if (currentTag.startsWith('news:') && !currentItem.news) {
currentItem.news = {
publication: { name: "", language: "" },
// eslint-disable-next-line @typescript-eslint/camelcase
publication_date: "",
title: ""
};
currentItem.news = newsTemplate();
}
});
saxStream.on('opentag', (tag) => {
this.saxStream.on('opentag', (tag) => {
switch (tag.name) {

@@ -121,24 +126,21 @@ case "url":

case "news:language":
break;
case "mobile:mobile":
currentItem.mobile = true;
break;
case 'xhtml:link':
// @ts-ignore
if (typeof tag.attributes.rel === "string" ||
typeof tag.attributes.href === "string") {
break;
}
if (tag.attributes.rel.value === 'alternate' && tag.attributes.hreflang) {
// @ts-ignore
currentLink.url = tag.attributes.href.value;
// @ts-ignore
if (typeof tag.attributes.hreflang === 'string')
break;
currentLink.lang = tag.attributes.hreflang.value;
// @ts-ignore
}
else if (tag.attributes.rel.value === 'alternate') {
dontpushCurrentLink = true;
// @ts-ignore
currentItem.androidLink = tag.attributes.href.value;
// @ts-ignore
}
else if (tag.attributes.rel.value === 'amphtml') {
dontpushCurrentLink = true;
// @ts-ignore
currentItem.ampLink = tag.attributes.href.value;

@@ -155,3 +157,3 @@ }

});
saxStream.on('text', (text) => {
this.saxStream.on('text', (text) => {
switch (currentTag) {

@@ -244,11 +246,17 @@ case "mobile:mobile":

case "news:access":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.access = text;
break;
case "news:genres":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.genres = text;
break;
case "news:publication_date":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
// eslint-disable-next-line @typescript-eslint/camelcase

@@ -258,7 +266,11 @@ currentItem.news.publication_date = text;

case "news:keywords":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.keywords = text;
break;
case "news:stock_tickers":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
// eslint-disable-next-line @typescript-eslint/camelcase

@@ -268,5 +280,41 @@ currentItem.news.stock_tickers = text;

case "news:language":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.publication.language = text;
break;
case "video:title":
currentVideo.title += text;
break;
case "video:description":
currentVideo.description += text;
break;
case "news:name":
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.publication.name += text;
break;
case "news:title":
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.title += text;
break;
case "image:caption":
if (!currentImage.caption) {
currentImage.caption = text;
}
else {
currentImage.caption += text;
}
break;
case "image:title":
if (!currentImage.title) {
currentImage.title = text;
}
else {
currentImage.title += text;
}
break;
default:

@@ -277,3 +325,3 @@ console.log('unhandled text for tag:', currentTag, `'${text}'`);

});
saxStream.on('cdata', (text) => {
this.saxStream.on('cdata', (text) => {
switch (currentTag) {

@@ -287,7 +335,11 @@ case "video:title":

case "news:name":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.publication.name += text;
break;
case "news:title":
// @ts-ignore
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
currentItem.news.title += text;

@@ -316,3 +368,3 @@ break;

});
saxStream.on('attribute', (attr) => {
this.saxStream.on('attribute', (attr) => {
switch (currentTag) {

@@ -373,11 +425,11 @@ case "urlset":

});
saxStream.on('closetag', (tag) => {
this.saxStream.on('closetag', (tag) => {
switch (tag) {
case 'url':
smi.push(currentItem);
currentItem = Object.assign({}, tagTemplate, { video: [], img: [], links: [] });
this.push(currentItem);
currentItem = tagTemplate();
break;
case "video:video":
currentItem.video.push(currentVideo);
currentVideo = Object.assign({}, videoTemplate, { tag: [] });
currentVideo = videoTemplate();
break;

@@ -398,8 +450,39 @@ case "image:image":

});
}
_transform(data, encoding, callback) {
this.saxStream.write(data, encoding);
callback();
}
}
exports.XMLToISitemapOptions = XMLToISitemapOptions;
/**
Read xml and resolve with the configuration that would produce it or reject with
an error
```
const { createReadStream } = require('fs')
const { parseSitemap, createSitemap } = require('sitemap')
parseSitemap(createReadStream('./example.xml')).then(
// produces the same xml
// you can, of course, more practically modify it or store it
(xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
(err) => console.log(err)
)
```
@param {Readable} xml what to parse
@return {Promise<ISitemapOptions>} resolves with a valid config that can be
passed to createSitemap. Rejects with an Error object.
*/
function parseSitemap(xml) {
return __awaiter(this, void 0, void 0, function* () {
// eslint-disable-next-line @typescript-eslint/ban-ts-ignore
// @ts-ignore
const urls = [];
return new Promise((resolve, reject) => {
saxStream.on('end', () => {
resolve({ urls: smi });
});
xml.pipe(saxStream);
saxStream.on('error', (error) => {
xml
.pipe(new XMLToISitemapOptions())
.on("data", (smi) => urls.push(smi))
.on("end", () => {
resolve({ urls });
})
.on("error", (error) => {
reject(error);

@@ -411,2 +494,43 @@ });

exports.parseSitemap = parseSitemap;
const defaultObjectStreamOpts = {
lineSeparated: false
};
/**
* A Transform that converts a stream of objects into a JSON Array or a line
* separated stringified JSON
* @param [lineSeparated=false] whether to separate entries by a new line or comma
*/
class ObjectStreamToJSON extends stream_1.Transform {
constructor(opts = defaultObjectStreamOpts) {
opts.writableObjectMode = true;
super(opts);
this.lineSeparated = opts.lineSeparated;
this.firstWritten = false;
}
_transform(chunk, encoding, cb) {
if (!this.firstWritten) {
this.firstWritten = true;
if (!this.lineSeparated) {
this.push('[');
}
}
else if (this.lineSeparated) {
this.push('\n');
}
else {
this.push(',');
}
if (chunk) {
this.push(JSON.stringify(chunk));
}
cb();
}
_flush(cb) {
if (!this.lineSeparated) {
this.push(']');
}
cb();
}
}
exports.ObjectStreamToJSON = ObjectStreamToJSON;
//# sourceMappingURL=sitemap-parser.js.map

34

dist/lib/sitemap.d.ts

@@ -18,15 +18,2 @@ /// <reference types="node" />

}
/**
* Shortcut for `new Sitemap (...)`.
*
* @param {Object} conf
* @param {String} conf.hostname
* @param {String|Array} conf.urls
* @param {Number} conf.cacheTime
* @param {String} conf.xslUrl
* @param {String} conf.xmlNs
* @param {ErrorLevel} [level=ErrorLevel.WARN] level optional
* @return {Sitemap}
*/
export declare function createSitemap({ urls, hostname, cacheTime, xslUrl, xmlNs, level }: ISitemapOptions): Sitemap;
export declare class Sitemap {

@@ -44,2 +31,4 @@ limit: number;

* Sitemap constructor
* @deprecated This API will go away in the next major release - use streamToPromise
* & SitemapStream
* @param {String|Array} urls

@@ -96,15 +85,13 @@ * @param {String} hostname optional

* @param {string | ISitemapItemOptionsLoose} elem the string or object to be converted
* @param {XMLElement=} root xmlbuilder root object. Pass undefined here
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
static normalizeURL(elem: string | ISitemapItemOptionsLoose, root?: XMLElement, hostname?: string): SitemapItemOptions;
static normalizeURL(elem: string | ISitemapItemOptionsLoose, hostname?: string): SitemapItemOptions;
/**
* Normalize multiple urls
* @param {(string | ISitemapItemOptionsLoose)[]} urls array of urls to be normalized
* @param {XMLElement=} root xmlbuilder root object. Pass undefined here
* @param {string=} hostname
* @returns a Map of url to SitemapItemOption
*/
static normalizeURLs(urls: (string | ISitemapItemOptionsLoose)[], root?: XMLElement, hostname?: string): Map<string, SitemapItemOptions>;
static normalizeURLs(urls: (string | ISitemapItemOptionsLoose)[], hostname?: string): Map<string, SitemapItemOptions>;
/**

@@ -127,1 +114,14 @@ * Converts the urls stored in an instance of Sitemap to a valid sitemap xml document

}
/**
* Shortcut for `new Sitemap (...)`.
*
* @param {Object} conf
* @param {String} conf.hostname
* @param {String|Array} conf.urls
* @param {Number} conf.cacheTime
* @param {String} conf.xslUrl
* @param {String} conf.xmlNs
* @param {ErrorLevel} [level=ErrorLevel.WARN] level optional
* @return {Sitemap}
*/
export declare function createSitemap({ urls, hostname, cacheTime, xslUrl, xmlNs, level }: ISitemapOptions): Sitemap;

96

dist/lib/sitemap.js

@@ -16,2 +16,3 @@ "use strict";

const utils_1 = require("./utils");
const sitemap_stream_1 = require("./sitemap-stream");
function boolToYESNO(bool) {

@@ -26,30 +27,7 @@ if (bool === undefined) {

}
/**
* Shortcut for `new Sitemap (...)`.
*
* @param {Object} conf
* @param {String} conf.hostname
* @param {String|Array} conf.urls
* @param {Number} conf.cacheTime
* @param {String} conf.xslUrl
* @param {String} conf.xmlNs
* @param {ErrorLevel} [level=ErrorLevel.WARN] level optional
* @return {Sitemap}
*/
function createSitemap({ urls, hostname, cacheTime, xslUrl, xmlNs, level }) {
// cleaner diff
// eslint-disable-next-line @typescript-eslint/no-use-before-define
return new Sitemap({
urls,
hostname,
cacheTime,
xslUrl,
xmlNs,
level
});
}
exports.createSitemap = createSitemap;
class Sitemap {
/**
* Sitemap constructor
* @deprecated This API will go away in the next major release - use streamToPromise
* & SitemapStream
* @param {String|Array} urls

@@ -78,3 +56,3 @@ * @param {String} hostname optional

const ns = this.xmlNs.split(' ');
for (let attr of ns) {
for (const attr of ns) {
const [k, v] = attr.split('=');

@@ -85,4 +63,4 @@ this.root.attribute(k, v.replace(/^['"]|['"]$/g, ''));

urls = Array.from(urls);
this.urls = Sitemap.normalizeURLs(urls, this.root, this.hostname);
for (let [, url] of this.urls) {
this.urls = Sitemap.normalizeURLs(urls, this.hostname);
for (const [, url] of this.urls) {
utils_1.validateSMIOptions(url, level);

@@ -102,3 +80,3 @@ }

isCacheValid() {
let currTimestamp = Date.now();
const currTimestamp = Date.now();
return !!(this.cacheTime && this.cache &&

@@ -119,3 +97,3 @@ (this.cacheSetTimestamp + this.cacheTime) >= currTimestamp);

_normalizeURL(url) {
return Sitemap.normalizeURL(url, this.root, this.hostname);
return Sitemap.normalizeURL(url, this.hostname);
}

@@ -158,7 +136,6 @@ /**

* @param {string | ISitemapItemOptionsLoose} elem the string or object to be converted
* @param {XMLElement=} root xmlbuilder root object. Pass undefined here
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
static normalizeURL(elem, root, hostname) {
static normalizeURL(elem, hostname) {
// SitemapItem

@@ -194,3 +171,3 @@ // create object with url property

// prepend hostname to all image urls
smi.img = img.map((el) => (Object.assign({}, el, { url: (new url_1.URL(el.url, hostname)).toString() })));
smi.img = img.map((el) => (Object.assign(Object.assign({}, el), { url: (new url_1.URL(el.url, hostname)).toString() })));
let links = [];

@@ -201,3 +178,3 @@ if (smiLoose.links) {

smi.links = links.map((link) => {
return Object.assign({}, link, { url: (new url_1.URL(link.url, hostname)).toString() });
return Object.assign(Object.assign({}, link), { url: (new url_1.URL(link.url, hostname)).toString() });
});

@@ -210,3 +187,3 @@ if (smiLoose.video) {

smi.video = smiLoose.video.map((video) => {
const nv = Object.assign({}, video, {
const nv = Object.assign(Object.assign({}, video), {
/* eslint-disable-next-line @typescript-eslint/camelcase */

@@ -246,3 +223,5 @@ family_friendly: boolToYESNO(video.family_friendly), live: boolToYESNO(video.live),

}
smi = Object.assign({}, smiLoose, smi);
delete smiLoose.lastmodfile;
delete smiLoose.lastmodISO;
smi = Object.assign(Object.assign({}, smiLoose), smi);
return smi;

@@ -253,10 +232,9 @@ }

* @param {(string | ISitemapItemOptionsLoose)[]} urls array of urls to be normalized
* @param {XMLElement=} root xmlbuilder root object. Pass undefined here
* @param {string=} hostname
* @returns a Map of url to SitemapItemOption
*/
static normalizeURLs(urls, root, hostname) {
static normalizeURLs(urls, hostname) {
const urlMap = new Map();
urls.forEach((elem) => {
const smio = Sitemap.normalizeURL(elem, root, hostname);
const smio = Sitemap.normalizeURL(elem, hostname);
urlMap.set(smio.url, smio);

@@ -273,2 +251,13 @@ });

toString(pretty = false) {
if (this.isCacheValid()) {
return this.cache;
}
if (this.urls && !this.xslUrl && !this.xmlNs && !pretty) {
let xml = sitemap_stream_1.preamble;
this.urls.forEach((url) => {
xml += sitemap_item_1.SitemapItem.justItem(url);
});
xml += sitemap_stream_1.closetag;
return this.setCache(xml);
}
if (this.root.children.length) {

@@ -281,3 +270,2 @@ this.root.children = [];

this.root.att('xmlns:xhtml', 'http://www.w3.org/1999/xhtml');
this.root.att('xmlns:mobile', 'http://www.google.com/schemas/sitemap-mobile/1.0');
this.root.att('xmlns:image', 'http://www.google.com/schemas/sitemap-image/1.1');

@@ -289,7 +277,4 @@ this.root.att('xmlns:video', 'http://www.google.com/schemas/sitemap-video/1.1');

}
if (this.isCacheValid()) {
return this.cache;
}
// TODO: if size > limit: create sitemapindex
for (let [, smi] of this.urls) {
for (const [, smi] of this.urls) {
(new sitemap_item_1.SitemapItem(smi, this.root)).buildXML();

@@ -313,2 +298,25 @@ }

exports.Sitemap = Sitemap;
/**
* Shortcut for `new Sitemap (...)`.
*
* @param {Object} conf
* @param {String} conf.hostname
* @param {String|Array} conf.urls
* @param {Number} conf.cacheTime
* @param {String} conf.xslUrl
* @param {String} conf.xmlNs
* @param {ErrorLevel} [level=ErrorLevel.WARN] level optional
* @return {Sitemap}
*/
function createSitemap({ urls, hostname, cacheTime, xslUrl, xmlNs, level }) {
return new Sitemap({
urls,
hostname,
cacheTime,
xslUrl,
xmlNs,
level
});
}
exports.createSitemap = createSitemap;
//# sourceMappingURL=sitemap.js.map

1

dist/lib/types.d.ts

@@ -101,3 +101,2 @@ /// <reference types="node" />

androidLink?: string;
mobile?: boolean | string;
ampLink?: string;

@@ -104,0 +103,0 @@ url: string;

28

dist/lib/utils.d.ts

@@ -6,5 +6,33 @@ /*!

*/
/// <reference types="node" />
import { SitemapItemOptions, ErrorLevel } from './types';
import { Readable, ReadableOptions } from 'stream';
export declare function validateSMIOptions(conf: SitemapItemOptions, level?: ErrorLevel): SitemapItemOptions;
/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
export declare function mergeStreams(streams: Readable[]): Readable;
export interface IReadLineStreamOptions extends ReadableOptions {
input: Readable;
}
/**
* Wraps node's ReadLine in a stream
*/
export declare class ReadLineStream extends Readable {
private _source;
constructor(options: IReadLineStreamOptions);
_read(size: number): void;
}
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
export declare function lineSeparatedURLsToSitemapOptions(stream: Readable, { isJSON }?: {
isJSON?: boolean;
}): Readable;
/**
* Based on lodash's implementation of chunk.

@@ -11,0 +39,0 @@ *

117

dist/lib/utils.js

@@ -10,2 +10,4 @@ "use strict";

const errors_1 = require("./errors");
const stream_1 = require("stream");
const readline_1 = require("readline");
const allowDeny = /^allow|deny$/;

@@ -17,4 +19,24 @@ const validators = {

'platform:relationship': allowDeny,
'restriction:relationship': allowDeny
'restriction:relationship': allowDeny,
'restriction': /^([A-Z]{2}( +[A-Z]{2})*)?$/,
'platform': /^((web|mobile|tv)( (web|mobile|tv))*)?$/,
'language': /^zh-cn|zh-tw|([a-z]{2,3})$/,
'genres': /^(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated)(, *(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated))*$/,
'stock_tickers': /^(\w+:\w+(, *\w+:\w+){0,4})?$/,
};
function validate(subject, name, url, level) {
Object.keys(subject).forEach((key) => {
// eslint-disable-next-line @typescript-eslint/ban-ts-ignore
// @ts-ignore
const val = subject[key];
if (validators[key] && !validators[key].test(val)) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidAttrValue(key, val, validators[key]);
}
else {
console.warn(`${url}: ${name} key ${key} has invalid value: ${val}`);
}
}
});
}
function validateSMIOptions(conf, level = types_1.ErrorLevel.WARN) {

@@ -47,3 +69,3 @@ if (!conf) {

if (priority) {
if (!(priority >= 0.0 && priority <= 1.0) || typeof priority !== 'number') {
if (!(priority >= 0.0 && priority <= 1.0)) {
if (level === types_1.ErrorLevel.THROW) {

@@ -80,2 +102,4 @@ throw new errors_1.PriorityInvalidError();

}
validate(news, 'news', url, level);
validate(news.publication, 'publication', url, level);
}

@@ -95,3 +119,8 @@ if (video) {

if (vid.rating !== undefined && (vid.rating < 0 || vid.rating > 5)) {
console.warn(`${url}: video ${vid.title} rating ${vid.rating} must be between 0 and 5 inclusive`);
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidVideoRating();
}
else {
console.warn(`${url}: video ${vid.title} rating ${vid.rating} must be between 0 and 5 inclusive`);
}
}

@@ -115,15 +144,3 @@ if (typeof (vid) !== 'object' || !vid.thumbnail_loc || !vid.title || !vid.description) {

}
Object.keys(vid).forEach((key) => {
// @ts-ignore
if (validators[key] && !validators[key].test(vid[key])) {
if (level === types_1.ErrorLevel.THROW) {
// @ts-ignore
throw new errors_1.InvalidAttrValue(key, vid[key], validators[key]);
}
else {
// @ts-ignore
console.warn(`${url}: video key ${key} has invalid value: ${vid[key]}`);
}
}
});
validate(vid, 'video', url, level);
});

@@ -135,2 +152,70 @@ }

/**
* Combines multiple streams into one
* @param streams the streams to combine
*/
function mergeStreams(streams) {
let pass = new stream_1.PassThrough();
let waiting = streams.length;
for (const stream of streams) {
pass = stream.pipe(pass, { end: false });
stream.once('end', () => --waiting === 0 && pass.emit('end'));
}
return pass;
}
exports.mergeStreams = mergeStreams;
/**
* Wraps node's ReadLine in a stream
*/
class ReadLineStream extends stream_1.Readable {
constructor(options) {
if (options.autoDestroy === undefined) {
options.autoDestroy = true;
}
options.objectMode = true;
super(options);
this._source = readline_1.createInterface({
input: options.input,
terminal: false,
crlfDelay: Infinity
});
// Every time there's data, push it into the internal buffer.
this._source.on('line', (chunk) => {
// If push() returns false, then stop reading from source.
if (!this.push(chunk))
this._source.pause();
});
// When the source ends, push the EOF-signaling `null` chunk.
this._source.on('close', () => {
this.push(null);
});
}
// _read() will be called when the stream wants to pull more data in.
// The advisory size argument is ignored in this case.
_read(size) {
this._source.resume();
}
}
exports.ReadLineStream = ReadLineStream;
/**
* Takes a stream likely from fs.createReadStream('./path') and returns a stream
* of sitemap items
* @param stream a stream of line separated urls.
* @param opts
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
function lineSeparatedURLsToSitemapOptions(stream, { isJSON } = {}) {
return new ReadLineStream({ input: stream }).pipe(new stream_1.Transform({
objectMode: true,
transform: (line, encoding, cb) => {
if (isJSON || (isJSON === undefined && line[0] === "{")) {
cb(null, JSON.parse(line));
}
else {
cb(null, line);
}
}
}));
}
exports.lineSeparatedURLsToSitemapOptions = lineSeparatedURLsToSitemapOptions;
/**
* Based on lodash's implementation of chunk.

@@ -137,0 +222,0 @@ *

7

dist/lib/xmllint.js

@@ -11,3 +11,3 @@ "use strict";

function xmlLint(xml) {
let args = ['--schema', './schema/all.xsd', '--noout', '-'];
const args = ['--schema', './schema/all.xsd', '--noout', '-'];
if (typeof xml === 'string') {

@@ -22,5 +22,4 @@ args[args.length - 1] = xml;

}
let xmllint = child_process_1.execFile('xmllint', args, (error, stdout, stderr) => {
// @ts-ignore
if (error && error.code) {
const xmllint = child_process_1.execFile('xmllint', args, (error, stdout, stderr) => {
if (error) {
reject([error, stderr]);

@@ -27,0 +26,0 @@ }

70

package.json
{
"name": "sitemap",
"version": "4.1.1",
"version": "5.0.0",
"description": "Sitemap-generating lib/cli",

@@ -30,7 +30,11 @@ "keywords": [

"test": "eslint lib/* ./cli.ts && tsc && jest && npm run test:xmllint",
"test-fast": "jest ./tests/sitemap-item.test.ts ./tests/sitemap-index.test.ts ./tests/sitemap.test.ts ./tests/sitemap-shape.test.ts",
"test-perf": "node ./tests/perf.js > /dev/null",
"test:fast": "eslint lib/* ./cli.ts && tsc && jest ./tests/sitemap*",
"test:perf": "node ./tests/perf.js",
"test:schema": "node tests/alltags.js | xmllint --schema schema/all.xsd --noout -",
"test:typecheck": "tsc",
"test:xmllint": "if which xmllint; then npm run test:schema; else echo 'skipping xml tests. xmllint not installed'; fi"
"test:xmllint": "if which xmllint; then npm run test:schema; else echo 'skipping xml tests. xmllint not installed'; fi",
"watch": "concurrently \"npm:watch:*\"",
"watch:eslint": "eslint -w lib/* ./cli.ts",
"watch:jest": "jest --watch ./tests/sitemap*",
"watch:tsc": "tsc -w"
},

@@ -40,3 +44,3 @@ "husky": {

"pre-commit": "sort-package-json",
"pre-push": "npm test"
"pre-push": "npm run test:fast"
}

@@ -47,3 +51,2 @@ },

"es6": true,
"jasmine": true,
"jest": true,

@@ -54,2 +57,3 @@ "node": true

"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended"

@@ -62,2 +66,6 @@ ],

},
"plugins": [
"jest",
"@typescript-eslint"
],
"rules": {

@@ -67,5 +75,19 @@ "no-case-declarations": 0,

"no-unused-vars": 0,
"react/prop-types": 0,
"indent": "off",
"no-dupe-class-members": "off",
"lines-between-class-members": [
"error",
"always",
{
"exceptAfterSingleLine": true
}
],
"padding-line-between-statements": [
"error",
{
"blankLine": "always",
"prev": "multiline-expression",
"next": "multiline-expression"
}
],
"@typescript-eslint/indent": [

@@ -107,3 +129,3 @@ "error",

"dependencies": {
"@types/node": "^12.0.2",
"@types/node": "^12.7.11",
"@types/sax": "^1.2.0",

@@ -115,23 +137,27 @@ "arg": "^4.1.1",

"devDependencies": {
"@babel/core": "^7.5.5",
"@babel/core": "^7.6.2",
"@babel/plugin-proposal-class-properties": "^7.5.5",
"@babel/plugin-transform-typescript": "^7.5.5",
"@babel/preset-env": "^7.5.5",
"@babel/preset-typescript": "^7.3.3",
"@types/jest": "^24.0.17",
"@typescript-eslint/eslint-plugin": "^1.13.0",
"@typescript-eslint/parser": "^1.13.0",
"babel-eslint": "^10.0.1",
"@babel/plugin-transform-typescript": "^7.6.0",
"@babel/preset-env": "^7.6.2",
"@babel/preset-typescript": "^7.6.0",
"@types/jest": "^24.0.18",
"@typescript-eslint/eslint-plugin": "^2.3.2",
"@typescript-eslint/parser": "^2.3.2",
"babel-eslint": "^10.0.3",
"babel-polyfill": "^6.26.0",
"eslint": "^6.1.0",
"husky": "^3.0.3",
"jasmine": "^3.4.0",
"jest": "^24.8.0",
"concurrently": "^4.1.2",
"eslint": "^6.5.1",
"eslint-plugin-jest": "^22.17.0",
"express": "^4.17.1",
"husky": "^3.0.8",
"jest": "^24.9.0",
"sort-package-json": "^1.22.1",
"source-map": "~0.7.3",
"stats-lite": "^2.2.0",
"typescript": "^3.5.3"
"stream-json": "^1.3.1",
"through2-map": "^3.0.0",
"typescript": "^3.6.3"
},
"engines": {
"node": ">=8.9.0",
"node": ">=10.0.0",
"npm": ">=5.6.0"

@@ -138,0 +164,0 @@ },

461

README.md
sitemap.js [![Build Status](https://travis-ci.org/ekalinin/sitemap.js.svg?branch=master)](https://travis-ci.org/ekalinin/sitemap.js)
==========
**sitemap.js** is a high-level sitemap-generating library/cli that
**sitemap.js** is a high-level sitemap-generating library/CLI that
makes creating [sitemap XML](http://www.sitemaps.org/) files easy.

@@ -20,4 +20,4 @@

* [CLI](#cli)
* [Example of using sitemap.js with <a href="https://expressjs.com/">express</a>:](#example-of-using-sitemapjs-with-express)
* [Example of dynamic page manipulations into sitemap:](#example-of-dynamic-page-manipulations-into-sitemap)
* [Example of using sitemap.js with <a href="https://expressjs.com/">express</a>](#example-of-using-sitemapjs-with-express)
* [Stream writing a sitemap](#stream-writing-a-sitemap)
* [Example of most of the options you can use for sitemap](#example-of-most-of-the-options-you-can-use-for-sitemap)

@@ -27,8 +27,12 @@ * [Building just the sitemap index file](#building-just-the-sitemap-index-file)

* [API](#api)
* [Create Sitemap](#create-sitemap)
* [Sitemap](#sitemap)
* [Sitemap (deprecated)](#sitemap---deprecated)
* [buildSitemapIndex](#buildsitemapindex)
* [createSitemapIndex](#createsitemapindex)
* [createSitemapsAndIndex](#createsitemapsandindex)
* [xmlLint](#xmllint)
* [parseSitemap](#parsesitemap)
* [SitemapStream](#sitemapstream)
* [XMLToISitemapOptions](#XMLToISitemapOptions)
* [lineSeparatedURLsToSitemapOptions](#lineseparatedurlstositemapoptions)
* [streamToPromise](#streamtopromise)
* [ObjectStreamToJSON](#objectstreamtojson)
* [Sitemap Item Options](#sitemap-item-options)

@@ -55,8 +59,4 @@ * [ISitemapImage](#isitemapimage)

Also supports line separated JSON for full configuration
Or verify an existing sitemap (requires libxml)
npx sitemap --json < listofurls.txt
Or verify an existing sitemap
npx sitemap --verify sitemap.xml

@@ -66,33 +66,51 @@

```javascript
const { createSitemap } = require('sitemap')
```js
const { SitemapStream, streamToPromise } = require('../dist/index')
// Creates a sitemap object given the input configuration with URLs
const sitemap = createSitemap({ options });
// Gives you a string containing the XML data
const xml = sitemap.toString();
const sitemap = new SitemapStream({ hostname: 'http://example.com' });
sitemap.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
sitemap.write('/page-2')
sitemap.end()
streamToPromise(sitemap)
.then(sm => console.log(sm.toString()))
.catch(console.error);
```
### Example of using sitemap.js with [express](https://github.com/visionmedia/express):
Resolves to a string containing the XML data
```xml
<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"><url><loc>http://example.com/page-1/</loc><changefreq>daily</changefreq><priority>0.3</priority></url><url><loc>http://example.com/page-2</loc></url></urlset>
```
```javascript
### Example of using sitemap.js with [express](https://expressjs.com/)
```js
const express = require('express')
const { createSitemap } = require('sitemap');
const { SitemapStream, streamToPromise } = require('sitemap')
const { createGzip } = require('zlib')
const app = express()
const sitemap = createSitemap({
hostname: 'http://example.com',
cacheTime: 600000, // 600 sec - cache purge period
urls: [
{ url: '/page-1/', changefreq: 'daily', priority: 0.3 },
{ url: '/page-2/', changefreq: 'monthly', priority: 0.7 },
{ url: '/page-3/'}, // changefreq: 'weekly', priority: 0.5
{ url: '/page-4/', img: "http://urlTest.com" }
]
});
let sitemap
app.get('/sitemap.xml', function(req, res) {
res.header('Content-Type', 'application/xml');
res.header('Content-Encoding', 'gzip');
// if we have a cached entry send it
if (sitemap) {
res.send(sitemap)
return
}
try {
const xml = sitemap.toXML()
res.header('Content-Type', 'application/xml');
res.send( xml );
const smStream = new SitemapStream({ hostname: 'https://example.com/' })
.pipe(createGzip())
smStream.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
smStream.write({ url: '/page-2/', changefreq: 'monthly', priority: 0.7 })
smStream.write({ url: '/page-3/'}) // changefreq: 'weekly', priority: 0.5
smStream.write({ url: '/page-4/', img: "http://urlTest.com" })
// cache the response
streamToPromise(gzippedStream).then(sm => sitemap = sm)
// stream the response
gzippedStream.pipe(res).on('error', (e) => {throw e})
} catch (e) {

@@ -102,104 +120,141 @@ console.error(e)

}
});
})
app.listen(3000, () => {
console.log('listening')
});
app.listen(3000);
```
### Example of dynamic page manipulations into sitemap:
### Stream writing a sitemap
The sitemap stream is around 20% faster and only uses ~10% the memory of the traditional interface
```javascript
const sitemap = createSitemap ({
hostname: 'http://example.com',
cacheTime: 600000
});
sitemap.add({url: '/page-1/'});
sitemap.add({url: '/page-2/', changefreq: 'monthly', priority: 0.7});
sitemap.del({url: '/page-2/'});
sitemap.del('/page-1/');
```
const fs = require('fs');
const { SitemapStream } = require('sitemap')
// external libs provided as example only
const { parser } = require('stream-json/Parser');
const { streamArray } = require('stream-json/streamers/StreamArray');
const { streamValues } = require('stream-json/streamers/StreamValues');
const map = require('through2-map')
const { createGzip } = require('zlib')
// parsing line separated json or JSONStream
const pipeline = fs
.createReadStream("./tests/mocks/perf-data.json.txt"),
.pipe(parser())
.pipe(streamValues())
.pipe(map.obj(chunk => chunk.value))
// SitemapStream does the heavy lifting
// You must provide it with an object stream
.pipe(new SitemapStream());
// parsing JSON file
const pipeline = fs
.createReadStream("./tests/mocks/perf-data.json")
.pipe(parser())
.pipe(streamArray())
.pipe(map.obj(chunk => chunk.value))
// SitemapStream does the heavy lifting
// You must provide it with an object stream
.pipe(new SitemapStream({ hostname: 'https://example.com/' }))
.pipe(process.stdout)
//
// coalesce into value for caching
//
let cachedXML
streamToPromise(
fs.createReadStream("./tests/mocks/perf-data.json")
.pipe(parser())
.pipe(streamArray())
.pipe(map.obj(chunk => chunk.value)),
.pipe(new SitemapStream({ hostname: 'https://example.com/' }))
.pipe(createGzip())
).then(xmlBuffer => cachedXML = xmlBuffer)
```
### Example of most of the options you can use for sitemap
```javascript
const { createSitemap } = require('sitemap');
```js
const { SitemapStream, streamToPromise } = require('sitemap');
const smStream = new SitemapStream({ hostname: 'http://www.mywebsite.com' })
// coalesce stream to value
// alternatively you can pipe to another stream
streamToSitemap(smStream).then(console.log)
const sitemap = createSitemap({
hostname: 'http://www.mywebsite.com',
level: 'warn', // default WARN about bad data
urls: [
smStream.write({
url: '/page1',
changefreq: 'weekly',
priority: 0.8,
lastmodfile: 'app/assets/page1.html'
})
smStream.write({
url: '/page2',
changefreq: 'weekly',
priority: 0.8,
/* useful to monitor template content files instead of generated static files */
lastmodfile: 'app/templates/page2.hbs'
})
// each sitemap entry supports many options
// See [Sitemap Item Options](#sitemap-item-options) below for details
smStream.write({
url: 'http://test.com/page-1/',
img: [
{
url: '/page1',
changefreq: 'weekly',
priority: 0.8,
lastmodfile: 'app/assets/page1.html'
url: 'http://test.com/img1.jpg',
caption: 'An image',
title: 'The Title of Image One',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
},
{
url: '/page2',
changefreq: 'weekly',
priority: 0.8,
/* useful to monitor template content files instead of generated static files */
lastmodfile: 'app/templates/page2.hbs'
url: 'http://test.com/img2.jpg',
caption: 'Another image',
title: 'The Title of Image Two',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
}
],
video: [
{
thumbnail_loc: 'http://test.com/tmbn1.jpg',
title: 'A video title',
description: 'This is a video'
},
// each sitemap entry supports many options
// See [Sitemap Item Options](#sitemap-item-options) below for details
{
url: 'http://test.com/page-1/',
img: [
{
url: 'http://test.com/img1.jpg',
caption: 'An image',
title: 'The Title of Image One',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
},
{
url: 'http://test.com/img2.jpg',
caption: 'Another image',
title: 'The Title of Image Two',
geoLocation: 'London, United Kingdom',
license: 'https://creativecommons.org/licenses/by/4.0/'
}
],
video: [
{
thumbnail_loc: 'http://test.com/tmbn1.jpg',
title: 'A video title',
description: 'This is a video'
},
{
thumbnail_loc: 'http://test.com/tmbn2.jpg',
title: 'A video with an attribute',
description: 'This is another video',
'player_loc': 'http://www.example.com/videoplayer.mp4?video=123',
'player_loc:autoplay': 'ap=1'
}
],
links: [
{ lang: 'en', url: 'http://test.com/page-1/' },
{ lang: 'ja', url: 'http://test.com/page-1/ja/' }
],
androidLink: 'android-app://com.company.test/page-1/',
news: {
publication: {
name: 'The Example Times',
language: 'en'
},
genres: 'PressRelease, Blog',
publication_date: '2008-12-23',
title: 'Companies A, B in Merger Talks',
keywords: 'business, merger, acquisition, A, B',
stock_tickers: 'NASDAQ:A, NASDAQ:B'
}
thumbnail_loc: 'http://test.com/tmbn2.jpg',
title: 'A video with an attribute',
description: 'This is another video',
'player_loc': 'http://www.example.com/videoplayer.mp4?video=123',
'player_loc:autoplay': 'ap=1'
}
]
});
],
links: [
{ lang: 'en', url: 'http://test.com/page-1/' },
{ lang: 'ja', url: 'http://test.com/page-1/ja/' }
],
androidLink: 'android-app://com.company.test/page-1/',
news: {
publication: {
name: 'The Example Times',
language: 'en'
},
genres: 'PressRelease, Blog',
publication_date: '2008-12-23',
title: 'Companies A, B in Merger Talks',
keywords: 'business, merger, acquisition, A, B',
stock_tickers: 'NASDAQ:A, NASDAQ:B'
}
})
// indicate there is nothing left to write
smStream.end()
```
### Building just the sitemap index file
The sitemap index file merely points to other sitemaps
```javascript
```js
const { buildSitemapIndex } = require('sitemap')

@@ -209,3 +264,3 @@ const smi = buildSitemapIndex({

xslUrl: 'https://example.com/style.xsl' // optional
});
})
```

@@ -215,6 +270,5 @@

```javascript
const { createSitemapIndex } = require('sitemap')
const smi = createSitemapIndex({
cacheTime: 600000,
```js
const { createSitemapsAndIndex } = require('sitemap')
const smi = createSitemapsAndIndex({
hostname: 'http://www.sitemap.org',

@@ -225,18 +279,16 @@ sitemapName: 'sm-test',

urls: ['http://ya.ru', 'http://ya2.ru']
// optional:
// callback: function(err, result) {}
});
})
```
## API
### Sitemap - __deprecated__
### Sitemap
```
```js
const { Sitemap } = require('sitemap')
const smi = new Sitemap({
urls: [{url: '/path'}],
const sm = new Sitemap({
urls: [{ url: '/path' }],
hostname: 'http://example.com',
cacheTime: 0, // default
level: 'warn' // default warns if it encounters bad data
level: 'warn' // default warns if it encounters bad data
})

@@ -247,5 +299,5 @@ sm.toString() // returns the xml as a string

__toString__
```js
sm.toString(true)
```
smi.toString(true)
```
Converts the urls stored in an instance of Sitemap to a valid sitemap xml document as a string. Accepts a boolean as its first argument to designate on whether to pretty print. Defaults to false.

@@ -257,65 +309,68 @@

__toGzip__
```js
sm.toGzip ((xmlGzippedBuffer) => console.log(xmlGzippedBuffer))
sm.toGzip()
```
smi.toGzip ((xmlGzippedBuffer) => console.log(xmlGzippedBuffer));
smi.toGzip();
```
like toString, it builds the xmlDocument, then it runs gzip on the resulting string and returns it as a Buffer via callback or direct invokation
Like toString, it builds the xmlDocument, then it runs gzip on the resulting string and returns it as a Buffer via callback or direct invocation
__clearCache__
```js
sm.clearCache()
```
smi.clearCache()
```
cache will be emptied and will be bipassed until set again
Cache will be emptied and will be bypassed until set again
__isCacheValid__
```js
sm.isCacheValid()
```
smi.isCacheValid()
```
returns true if it has been less than cacheTimeout ms since cache was set
Returns true if it has been less than cacheTimeout ms since cache was set
__setCache__
```js
sm.setCache('...xmlDoc')
```
smi.setCache('...xmlDoc')
```
stores the passed in string on the instance to be used when toString is called within the configured cacheTimeout
Stores the passed in string on the instance to be used when toString is called within the configured cacheTimeout
returns the passed in string unaltered
__add__
```js
sm.add('/path', 'warn')
```
smi.add('/path', 'warn')
```
adds the provided url to the sitemap instance
takes an optional parameter level for whether to print a console warning in the event of bad data 'warn' (default), throw an exception 'throw', or quietly ignore bad data 'silent'
Adds the provided url to the sitemap instance
takes an optional parameter level for whether to print a console warning in the event of bad data 'warn' (default),
throw an exception 'throw', or quietly ignore bad data 'silent'
returns the number of locations currently in the sitemap instance
__contains__
```js
sm.contains('/path')
```
smi.contains('/path')
```
Returns true if path is already a part of the sitemap instance, false otherwise.
Returns true if path is already a part of the sitemap instance, false otherwise.
__del__
```js
sm.del('/path')
```
smi.del('/path')
```
removes the provided url or url option from the sitemap instance
Removes the provided url or url option from the sitemap instance
__normalizeURL__
```js
Sitemap.normalizeURL('/', 'http://example.com')
```
Sitemap.normalizeURL('/', undefined, 'http://example.com')
```
static function that returns the stricter form of a options passed to SitemapItem
Static function that returns the stricter form of a options passed to SitemapItem
__normalizeURLs__
```
```js
Sitemap.normalizeURLs(['http://example.com', {url: 'http://example.com'}])
```
static function that takes an array of urls and returns a Map of their resolved url to the strict form of SitemapItemOptions
Static function that takes an array of urls and returns a Map of their resolved url to the strict form of SitemapItemOptions
### buildSitemapIndex
Build a sitemap index file
```
```js
const { buildSitemapIndex } = require('sitemap')
const index = buildSitemapIndex({
urls: [{url: 'http://example.com/sitemap-1.xml', lastmod: '2019-07-01'}, 'http://example.com/sitemap-2.xml'],
const index = buildSitemapIndex({
urls: [{ url: 'http://example.com/sitemap-1.xml', lastmod: '2019-07-01' }, 'http://example.com/sitemap-2.xml'],
lastmod: '2019-07-29'

@@ -325,8 +380,7 @@ })

### createSitemapIndex
### createSitemapsAndIndex
Create several sitemaps and an index automatically from a list of urls
```
const { createSitemapIndex } = require('sitemap')
createSitemapIndex({
urls: [/* list of urls */],
```js
const { createSitemapsAndIndex } createsitemapsandindex('sitemap')
createSitemapsAndIndex(createsitemapsandindex: [/* list of urls */],
targetFolder: 'absolute path to target folder',

@@ -337,5 +391,3 @@ hostname: 'http://example.com',

sitemapSize: 50000, // number of urls to allow in each sitemap
xslUrl: '',// custom xsl url
gzip: false, // whether to gzip the files
callback: // called when complete;
gzip: true, // whether to gzip the files
})

@@ -345,6 +397,8 @@ ```

### xmlLint
Resolve or reject depending on whether the passed in xml is a valid sitemap.
This is just a wrapper around the xmllint command line tool and thus requires
xmllint.
```
This is just a wrapper around the xmlLint command line tool and thus requires
xmlLint.
```js
const { createReadStream } = require('fs')

@@ -359,5 +413,7 @@ const { xmlLint } = require('sitemap')

### parseSitemap
Read xml and resolve with the configuration that would produce it or reject with
an error
```
```js
const { createReadStream } = require('fs')

@@ -373,2 +429,53 @@ const { parseSitemap, createSitemap } = require('sitemap')

### SitemapStream
A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream) for turning a [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams) of either [SitemapItemOptions](#sitemap-item-options) or url strings into a Sitemap. The readable stream it transforms **must** be in object mode.
```javascript
const { SitemapStream } = require('sitemap')
const sms = new SitemapStream({
hostname: 'https://example.com' // optional only necessary if your paths are relative
})
const readable = // a readable stream of objects
readable.pipe(sms).pipe(process.stdout)
```
### XMLToISitemapOptions
Takes a stream of xml and transforms it into a stream of ISitemapOptions.
Use this to parse existing sitemaps into config options compatible with this library
```javascript
const { createReadStream, createWriteStream } = require('fs');
const { XMLToISitemapOptions, ObjectStreamToJSON } = require('sitemap');
createReadStream('./some/sitemap.xml')
// turn the xml into sitemap option item options
.pipe(new XMLToISitemapOptions())
// convert the object stream to JSON
.pipe(new ObjectStreamToJSON())
// write the library compatible options to disk
.pipe(createWriteStream('./sitemapOptions.json'))
```
### lineSeparatedURLsToSitemapOptions
Takes a stream of urls or sitemapoptions likely from fs.createReadStream('./path') and returns an object stream of sitemap items.
### streamToPromise
Takes a stream returns a promise that resolves when stream emits finish.
```javascript
const { streamToPromise, SitemapStream } = require('sitemap')
const sitemap = new SitemapStream({ hostname: 'http://example.com' });
sitemap.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
sitemap.end()
streamToPromise(sitemap).then(buffer => console.log(buffer.toString())) // emits the full sitemap
```
### ObjectStreamToJSON
A Transform that converts a stream of objects into a JSON Array or a line separated stringified JSON.
* @param [lineSeparated=false] whether to separate entries by a new line or comma
```javascript
const stream = Readable.from([{a: 'b'}])
.pipe(new ObjectStreamToJSON())
.pipe(process.stdout)
stream.end()
// prints {"a":"b"}
```
### Sitemap Item Options

@@ -379,3 +486,3 @@

|url|string|http://example.com/some/path|The only required property for every sitemap entry|
|lastmod|string|'2019-07-29' or '2019-07-22T05:58:37.037Z'|When the page we as last modified use the W3C Datetime ISO8601 subset https://www.sitemaps.org/protocol.html#xmlTagDefinitions|
|lastmod|string|'2019-07-29' or '2019-07-22T05:58:37.037Z'|When the page we as last modified use the W3C Datetime ISO8601 subset https://www.sitemaps.org/protocol.html#xmlTagDefinitions|
|changefreq|string|'weekly'|How frequently the page is likely to change. This value provides general information to search engines and may not correlate exactly to how often they crawl the page. Please note that the value of this tag is considered a hint and not a command. See https://www.sitemaps.org/protocol.html#xmlTagDefinitions for the acceptable values|

@@ -388,3 +495,2 @@ |priority|number|0.6|The priority of this URL relative to other URLs on your site. Valid values range from 0.0 to 1.0. This value does not affect how your pages are compared to pages on other sites—it only lets the search engines know which pages you deem most important for the crawlers. The default priority of a page is 0.5. https://www.sitemaps.org/protocol.html#xmlTagDefinitions|

|ampLink|string|'http://ampproject.org/article.amp.html'||
|mobile|boolean or string|||
|cdata|boolean|true|wrap url in cdata xml escape|

@@ -430,4 +536,4 @@

|price:type|string - optional|"rent"|type [Optional] Specifies the purchase option. Supported values are rent and own. |
|uploader|string - optional|"GrillyMcGrillerson"|The video uploader's name. Only one <video:uploader> is allowed per video. String value, max 255 charactersc.|
|platform|string - optional|"tv"|Whether to show or hide your video in search results on specified platform types. This is a list of space-delimited platform types. See https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190 for more detail|
|uploader|string - optional|"GrillyMcGrillerson"|The video uploader's name. Only one <video:uploader> is allowed per video. String value, max 255 characters.|
|platform|string - optional|"tv"|Whether to show or hide your video in search results on specified platform types. This is a list of space-delimited platform types. See https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190 for more detail|
|platform:relationship|string 'Allow'\|'Deny' - optional|'Allow'||

@@ -456,3 +562,3 @@ |id|string - optional|||

|------|----|--|-----------|
|access|string - 'Registration' \| 'Subscription'| 'Registration' - optional||
|access|string - 'Registration' \| 'Subscription'| 'Registration' - optional||
|publication| object|see following options||

@@ -470,3 +576,2 @@ |publication['name']| string|'The Example Times'|The <name> is the name of the news publication. It must exactly match the name as it appears on your articles on news.google.com, except for anything in parentheses.|

See [LICENSE](https://github.com/ekalinin/sitemap.js/blob/master/LICENSE)
file.
See [LICENSE](https://github.com/ekalinin/sitemap.js/blob/master/LICENSE) file.
dist/cli.js.map
dist/index.js.map
dist/lib/errors.js.map
dist/lib/sitemap-index.js.map
dist/lib/sitemap-item.js.map
dist/lib/sitemap-parser.js.map
dist/lib/sitemap.js.map
dist/lib/types.js.map
dist/lib/utils.js.map
dist/lib/xmllint.js.map
index.d.ts
try_lastmod.js
try_lastmod2.js
try.last-mod.js
schema/all.xsd

Sorry, the diff of this file is not supported yet

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

About

Packages

npm

Go

Maven

PyPI

Rubygems

Stay in touch

Get open source security insights delivered straight into your inbox.

  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc