Socket
Socket
Sign inDemoInstall

sitemap

Package Overview
Dependencies
Maintainers
1
Versions
73
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

sitemap - npm Package Compare versions

Comparing version 5.1.0 to 6.0.0

dist/lib/sitemap-index-stream.d.ts

23

CHANGELOG.md
# Changelog
## 6.0.0
- removed xmlbuilder as a dependency
- added stronger validity checking on values supplied to sitemap
- Added the ability to turn off or add custom xml namespaces
- CLI and library now can accept a stream which will automatically write both the index and the sitemaps. See README for usage.
### 6.0.0 breaking changes
- renamed XMLToISitemapOptions to XMLToSitemapOptions
- various error messages changed.
- removed deprecated Sitemap and SitemapIndex classes
- replaced buildSitemapIndex with SitemapIndexStream
- Typescript: various types renamed or made more specific, removed I prefix
- Typescript: view_count is now exclusively a number
- Typescript: `price:type` and `price:resolution` are now more restrictive types
- sitemap parser now returns a sitemapItem array rather than a config object that could be passed to the now removed Sitemap class
- CLI no longer accepts multiple file arguments or a mixture of file and streams except as a part of a parameter eg. prepend
## 5.1.0

@@ -37,3 +56,3 @@

### Breaking Changes
### 5.0 Breaking Changes

@@ -148,3 +167,3 @@ - Dropped support for mobile sitemap - Google appears to have deleted their dtd and all references to it, strongly implying that they do not want you to use it. As its absence now breaks the validator, it has been dropped.

### breaking changes
### 3.0 breaking changes

@@ -151,0 +170,0 @@ This will likely not break anyone's code but we're bumping to be safe

@@ -10,4 +10,15 @@ #!/usr/bin/env node

const sitemap_stream_1 = require("./lib/sitemap-stream");
const sitemap_index_stream_1 = require("./lib/sitemap-index-stream");
const url_1 = require("url");
const zlib_1 = require("zlib");
/* eslint-disable-next-line @typescript-eslint/no-var-requires */
const arg = require('arg');
const pickStreamOrArg = (argv) => {
if (!argv._.length) {
return process.stdin;
}
else {
return fs_1.createReadStream(argv._[0], { encoding: 'utf8' });
}
};
const argSpec = {

@@ -17,5 +28,10 @@ '--help': Boolean,

'--validate': Boolean,
'--index': Boolean,
'--index-base-url': String,
'--limit': Number,
'--parse': Boolean,
'--single-line-json': Boolean,
'--prepend': String,
'--gzip': Boolean,
'--h': '--help',
};

@@ -44,4 +60,8 @@ const argv = arg(argSpec);

--validate ensure the passed in file is conforms to the sitemap spec
--index create an index and stream that out, write out sitemaps along the way
--index-base-url base url the sitemaps will be hosted eg. https://example.com/sitemaps/
--limit=45000 set a custom limit to the items per sitemap
--parse Parse fed xml and spit out config
--prepend sitemap.xml < urlsToAdd.json
--gzip compress output
--single-line-json When used with parse, it spits out each entry as json rather

@@ -52,6 +72,9 @@ than the whole json.

else if (argv['--parse']) {
getStream()
.pipe(new sitemap_parser_1.XMLToISitemapOptions())
.pipe(new sitemap_parser_1.ObjectStreamToJSON({ lineSeparated: !argv['--single-line-json'] }))
.pipe(process.stdout);
let oStream = getStream()
.pipe(new sitemap_parser_1.XMLToSitemapItemStream())
.pipe(new sitemap_parser_1.ObjectStreamToJSON({ lineSeparated: !argv['--single-line-json'] }));
if (argv['--gzip']) {
oStream = oStream.pipe(zlib_1.createGzip());
}
oStream.pipe(process.stdout);
}

@@ -71,19 +94,42 @@ else if (argv['--validate']) {

}
else {
let streams;
if (!argv._.length) {
streams = [process.stdin];
else if (argv['--index']) {
const limit = argv['--limit'];
const baseURL = argv['--index-base-url'];
if (!baseURL) {
throw new Error("You must specify where the sitemaps will be hosted. use --index-base-url 'https://example.com/path'");
}
else {
streams = argv._.map((file) => fs_1.createReadStream(file, { encoding: 'utf8' }));
const sms = new sitemap_index_stream_1.SitemapAndIndexStream({
limit,
getSitemapStream: (i) => {
const sm = new sitemap_stream_1.SitemapStream();
const path = `./sitemap-${i}.xml`;
if (argv['--gzip']) {
sm.pipe(zlib_1.createGzip()).pipe(fs_1.createWriteStream(path));
}
else {
sm.pipe(fs_1.createWriteStream(path));
}
return [new url_1.URL(path, baseURL).toString(), sm];
},
});
let oStream = utils_1.lineSeparatedURLsToSitemapOptions(pickStreamOrArg(argv)).pipe(sms);
if (argv['--gzip']) {
oStream = oStream.pipe(zlib_1.createGzip());
}
oStream.pipe(process.stdout);
}
else {
const sms = new sitemap_stream_1.SitemapStream();
if (argv['--prepend']) {
fs_1.createReadStream(argv['--prepend'])
.pipe(new sitemap_parser_1.XMLToISitemapOptions())
.pipe(new sitemap_parser_1.XMLToSitemapItemStream())
.pipe(sms);
}
utils_1.lineSeparatedURLsToSitemapOptions(utils_1.mergeStreams(streams))
.pipe(sms)
.pipe(process.stdout);
const oStream = utils_1.lineSeparatedURLsToSitemapOptions(pickStreamOrArg(argv)).pipe(sms);
if (argv['--gzip']) {
oStream.pipe(zlib_1.createGzip()).pipe(process.stdout);
}
else {
oStream.pipe(process.stdout);
}
}

13

dist/index.d.ts

@@ -6,12 +6,9 @@ /*!

*/
import { createSitemap } from './lib/sitemap';
export * from './lib/sitemap';
export * from './lib/sitemap-item';
export * from './lib/sitemap-index';
export * from './lib/sitemap-stream';
export { SitemapItemStream, SitemapItemStreamOptions, } from './lib/sitemap-item-stream';
export { IndexTagNames, SitemapIndexStream, SitemapIndexStreamOptions, createSitemapsAndIndex, SitemapAndIndexStream, SitemapAndIndexStreamOptions, } from './lib/sitemap-index-stream';
export { streamToPromise, SitemapStream, SitemapStreamOptions, } from './lib/sitemap-stream';
export * from './lib/errors';
export * from './lib/types';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, validateSMIOptions, } from './lib/utils';
export { lineSeparatedURLsToSitemapOptions, mergeStreams, validateSMIOptions, normalizeURL, ReadlineStream, ReadlineStreamOptions, } from './lib/utils';
export { xmlLint } from './lib/xmllint';
export { parseSitemap, XMLToISitemapOptions, ObjectStreamToJSON, } from './lib/sitemap-parser';
export default createSitemap;
export { parseSitemap, XMLToSitemapItemStream, XMLToSitemapItemStreamOptions, ObjectStreamToJSON, ObjectStreamToJSONOptions, } from './lib/sitemap-parser';

@@ -11,7 +11,12 @@ "use strict";

*/
const sitemap_1 = require("./lib/sitemap");
__export(require("./lib/sitemap"));
__export(require("./lib/sitemap-item"));
__export(require("./lib/sitemap-index"));
__export(require("./lib/sitemap-stream"));
var sitemap_item_stream_1 = require("./lib/sitemap-item-stream");
exports.SitemapItemStream = sitemap_item_stream_1.SitemapItemStream;
var sitemap_index_stream_1 = require("./lib/sitemap-index-stream");
exports.IndexTagNames = sitemap_index_stream_1.IndexTagNames;
exports.SitemapIndexStream = sitemap_index_stream_1.SitemapIndexStream;
exports.createSitemapsAndIndex = sitemap_index_stream_1.createSitemapsAndIndex;
exports.SitemapAndIndexStream = sitemap_index_stream_1.SitemapAndIndexStream;
var sitemap_stream_1 = require("./lib/sitemap-stream");
exports.streamToPromise = sitemap_stream_1.streamToPromise;
exports.SitemapStream = sitemap_stream_1.SitemapStream;
__export(require("./lib/errors"));

@@ -23,2 +28,4 @@ __export(require("./lib/types"));

exports.validateSMIOptions = utils_1.validateSMIOptions;
exports.normalizeURL = utils_1.normalizeURL;
exports.ReadlineStream = utils_1.ReadlineStream;
var xmllint_1 = require("./lib/xmllint");

@@ -28,4 +35,3 @@ exports.xmlLint = xmllint_1.xmlLint;

exports.parseSitemap = sitemap_parser_1.parseSitemap;
exports.XMLToISitemapOptions = sitemap_parser_1.XMLToISitemapOptions;
exports.XMLToSitemapItemStream = sitemap_parser_1.XMLToSitemapItemStream;
exports.ObjectStreamToJSON = sitemap_parser_1.ObjectStreamToJSON;
exports.default = sitemap_1.createSitemap;

@@ -7,3 +7,3 @@ /*!

/**
* URL in SitemapItem does not exists
* URL in SitemapItem does not exist
*/

@@ -23,3 +23,3 @@ export declare class NoURLError extends Error {

export declare class ChangeFreqInvalidError extends Error {
constructor(message?: string);
constructor(url: string, changefreq: any);
}

@@ -30,3 +30,3 @@ /**

export declare class PriorityInvalidError extends Error {
constructor(message?: string);
constructor(url: string, priority: any);
}

@@ -40,12 +40,12 @@ /**

export declare class InvalidVideoFormat extends Error {
constructor(message?: string);
constructor(url: string);
}
export declare class InvalidVideoDuration extends Error {
constructor(message?: string);
constructor(url: string, duration: any);
}
export declare class InvalidVideoDescription extends Error {
constructor(message?: string);
constructor(url: string, length: number);
}
export declare class InvalidVideoRating extends Error {
constructor(message?: string);
constructor(url: string, title: any, rating: any);
}

@@ -59,6 +59,6 @@ export declare class InvalidAttrValue extends Error {

export declare class InvalidNewsFormat extends Error {
constructor(message?: string);
constructor(url: string);
}
export declare class InvalidNewsAccessValue extends Error {
constructor(message?: string);
constructor(url: string, access: any);
}

@@ -68,1 +68,31 @@ export declare class XMLLintUnavailable extends Error {

}
export declare class InvalidVideoTitle extends Error {
constructor(url: string, length: number);
}
export declare class InvalidVideoViewCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoTagCount extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoCategory extends Error {
constructor(url: string, count: number);
}
export declare class InvalidVideoFamilyFriendly extends Error {
constructor(url: string, fam: string);
}
export declare class InvalidVideoRestriction extends Error {
constructor(url: string, code: string);
}
export declare class InvalidVideoRestrictionRelationship extends Error {
constructor(url: string, val?: string);
}
export declare class InvalidVideoPriceType extends Error {
constructor(url: string, priceType?: string, price?: string);
}
export declare class InvalidVideoResolution extends Error {
constructor(url: string, resolution: string);
}
export declare class InvalidVideoPriceCurrency extends Error {
constructor(url: string, currency: string);
}
"use strict";
/* eslint-disable @typescript-eslint/no-explicit-any */
/*!

@@ -9,3 +10,3 @@ * Sitemap

/**
* URL in SitemapItem does not exists
* URL in SitemapItem does not exist
*/

@@ -35,4 +36,4 @@ class NoURLError extends Error {

class ChangeFreqInvalidError extends Error {
constructor(message) {
super(message || 'changefreq is invalid');
constructor(url, changefreq) {
super(`${url}: changefreq "${changefreq}" is invalid`);
this.name = 'ChangeFreqInvalidError';

@@ -47,4 +48,4 @@ Error.captureStackTrace(this, ChangeFreqInvalidError);

class PriorityInvalidError extends Error {
constructor(message) {
super(message || 'priority is invalid');
constructor(url, priority) {
super(`${url}: priority "${priority}" must be a number between 0 and 1 inclusive`);
this.name = 'PriorityInvalidError';

@@ -67,5 +68,4 @@ Error.captureStackTrace(this, PriorityInvalidError);

class InvalidVideoFormat extends Error {
constructor(message) {
super(message ||
'must include thumbnail_loc, title and description fields for videos');
constructor(url) {
super(`${url} video must include thumbnail_loc, title and description fields for videos`);
this.name = 'InvalidVideoFormat';

@@ -77,4 +77,4 @@ Error.captureStackTrace(this, InvalidVideoFormat);

class InvalidVideoDuration extends Error {
constructor(message) {
super(message || 'duration must be an integer of seconds between 0 and 28800');
constructor(url, duration) {
super(`${url} duration "${duration}" must be an integer of seconds between 0 and 28800`);
this.name = 'InvalidVideoDuration';

@@ -86,4 +86,5 @@ Error.captureStackTrace(this, InvalidVideoDuration);

class InvalidVideoDescription extends Error {
constructor(message) {
super(message || 'description must be no longer than 2048 characters');
constructor(url, length) {
const message = `${url}: video description is too long ${length} vs limit of 2048 characters.`;
super(message);
this.name = 'InvalidVideoDescription';

@@ -95,4 +96,4 @@ Error.captureStackTrace(this, InvalidVideoDescription);

class InvalidVideoRating extends Error {
constructor(message) {
super(message || 'rating must be between 0 and 5');
constructor(url, title, rating) {
super(`${url}: video "${title}" rating "${rating}" must be between 0 and 5 inclusive`);
this.name = 'InvalidVideoRating';

@@ -129,5 +130,4 @@ Error.captureStackTrace(this, InvalidVideoRating);

class InvalidNewsFormat extends Error {
constructor(message) {
super(message ||
'must include publication, publication name, publication language, title, and publication_date for news');
constructor(url) {
super(`${url} News must include publication, publication name, publication language, title, and publication_date for news`);
this.name = 'InvalidNewsFormat';

@@ -139,5 +139,4 @@ Error.captureStackTrace(this, InvalidNewsFormat);

class InvalidNewsAccessValue extends Error {
constructor(message) {
super(message ||
'News access must be either Registration, Subscription or not be present');
constructor(url, access) {
super(`${url} News access "${access}" must be either Registration, Subscription or not be present`);
this.name = 'InvalidNewsAccessValue';

@@ -156,1 +155,83 @@ Error.captureStackTrace(this, InvalidNewsAccessValue);

exports.XMLLintUnavailable = XMLLintUnavailable;
class InvalidVideoTitle extends Error {
constructor(url, length) {
super(`${url}: video title is too long ${length} vs 100 character limit`);
this.name = 'InvalidVideoTitle';
Error.captureStackTrace(this, InvalidVideoTitle);
}
}
exports.InvalidVideoTitle = InvalidVideoTitle;
class InvalidVideoViewCount extends Error {
constructor(url, count) {
super(`${url}: video view count must be positive, view count was ${count}`);
this.name = 'InvalidVideoViewCount';
Error.captureStackTrace(this, InvalidVideoViewCount);
}
}
exports.InvalidVideoViewCount = InvalidVideoViewCount;
class InvalidVideoTagCount extends Error {
constructor(url, count) {
super(`${url}: video can have no more than 32 tags, this has ${count}`);
this.name = 'InvalidVideoTagCount';
Error.captureStackTrace(this, InvalidVideoTagCount);
}
}
exports.InvalidVideoTagCount = InvalidVideoTagCount;
class InvalidVideoCategory extends Error {
constructor(url, count) {
super(`${url}: video category can only be 256 characters but was passed ${count}`);
this.name = 'InvalidVideoCategory';
Error.captureStackTrace(this, InvalidVideoCategory);
}
}
exports.InvalidVideoCategory = InvalidVideoCategory;
class InvalidVideoFamilyFriendly extends Error {
constructor(url, fam) {
super(`${url}: video family friendly must be yes or no, was passed "${fam}"`);
this.name = 'InvalidVideoFamilyFriendly';
Error.captureStackTrace(this, InvalidVideoFamilyFriendly);
}
}
exports.InvalidVideoFamilyFriendly = InvalidVideoFamilyFriendly;
class InvalidVideoRestriction extends Error {
constructor(url, code) {
super(`${url}: video restriction must be one or more two letter country codes. Was passed "${code}"`);
this.name = 'InvalidVideoRestriction';
Error.captureStackTrace(this, InvalidVideoRestriction);
}
}
exports.InvalidVideoRestriction = InvalidVideoRestriction;
class InvalidVideoRestrictionRelationship extends Error {
constructor(url, val) {
super(`${url}: video restriction relationship must be either allow or deny. Was passed "${val}"`);
this.name = 'InvalidVideoRestrictionRelationship';
Error.captureStackTrace(this, InvalidVideoRestrictionRelationship);
}
}
exports.InvalidVideoRestrictionRelationship = InvalidVideoRestrictionRelationship;
class InvalidVideoPriceType extends Error {
constructor(url, priceType, price) {
super(priceType === undefined && price === ''
? `${url}: video priceType is required when price is not provided`
: `${url}: video price type "${priceType}" is not "rent" or "purchase"`);
this.name = 'InvalidVideoPriceType';
Error.captureStackTrace(this, InvalidVideoPriceType);
}
}
exports.InvalidVideoPriceType = InvalidVideoPriceType;
class InvalidVideoResolution extends Error {
constructor(url, resolution) {
super(`${url}: video price resolution "${resolution}" is not hd or sd`);
this.name = 'InvalidVideoResolution';
Error.captureStackTrace(this, InvalidVideoResolution);
}
}
exports.InvalidVideoResolution = InvalidVideoResolution;
class InvalidVideoPriceCurrency extends Error {
constructor(url, currency) {
super(`${url}: video price currency "${currency}" must be a three capital letter abbrieviation for the country currency`);
this.name = 'InvalidVideoPriceCurrency';
Error.captureStackTrace(this, InvalidVideoPriceCurrency);
}
}
exports.InvalidVideoPriceCurrency = InvalidVideoPriceCurrency;
/// <reference types="node" />
import { SAXStream } from 'sax';
import { Readable, Transform, TransformOptions, TransformCallback } from 'stream';
import { SitemapItemOptions, ErrorLevel } from './types';
import { ISitemapOptions } from './sitemap';
export interface ISitemapStreamParseOpts extends TransformOptions, Pick<ISitemapOptions, 'level'> {
import { SitemapItem, ErrorLevel } from './types';
export interface XMLToSitemapItemStreamOptions extends TransformOptions {
level?: ErrorLevel;
}
/**
* Takes a stream of xml and transforms it into a stream of ISitemapOptions
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*/
export declare class XMLToISitemapOptions extends Transform {
export declare class XMLToSitemapItemStream extends Transform {
level: ErrorLevel;
saxStream: SAXStream;
constructor(opts?: ISitemapStreamParseOpts);
constructor(opts?: XMLToSitemapItemStreamOptions);
_transform(data: string, encoding: string, callback: TransformCallback): void;

@@ -32,7 +32,6 @@ }

@param {Readable} xml what to parse
@return {Promise<ISitemapOptions>} resolves with a valid config that can be
passed to createSitemap. Rejects with an Error object.
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object.
*/
export declare function parseSitemap(xml: Readable): Promise<ISitemapOptions>;
export interface IObjectToStreamOpts extends TransformOptions {
export declare function parseSitemap(xml: Readable): Promise<SitemapItem[]>;
export interface ObjectStreamToJSONOptions extends TransformOptions {
lineSeparated: boolean;

@@ -48,5 +47,5 @@ }

firstWritten: boolean;
constructor(opts?: IObjectToStreamOpts);
_transform(chunk: SitemapItemOptions, encoding: string, cb: TransformCallback): void;
constructor(opts?: ObjectStreamToJSONOptions);
_transform(chunk: SitemapItem, encoding: string, cb: TransformCallback): void;
_flush(cb: TransformCallback): void;
}

@@ -6,5 +6,10 @@ "use strict";

Object.defineProperty(exports, "__esModule", { value: true });
/* eslint-disable @typescript-eslint/camelcase */
const sax_1 = __importDefault(require("sax"));
const stream_1 = require("stream");
const types_1 = require("./types");
function isValidTagName(tagName) {
// This only works because the enum name and value are the same
return tagName in types_1.TagNames;
}
function tagTemplate() {

@@ -21,3 +26,2 @@ return {

tag: [],
// eslint-disable-next-line @typescript-eslint/camelcase
thumbnail_loc: '',

@@ -38,3 +42,2 @@ title: '',

publication: { name: '', language: '' },
// eslint-disable-next-line @typescript-eslint/camelcase
publication_date: '',

@@ -45,7 +48,8 @@ title: '',

const defaultStreamOpts = {};
// TODO does this need to end with `options`
/**
* Takes a stream of xml and transforms it into a stream of ISitemapOptions
* Takes a stream of xml and transforms it into a stream of SitemapItems
* Use this to parse existing sitemaps into config options compatible with this library
*/
class XMLToISitemapOptions extends stream_1.Transform {
class XMLToSitemapItemStream extends stream_1.Transform {
constructor(opts = defaultStreamOpts) {

@@ -75,52 +79,7 @@ opts.objectMode = true;

this.saxStream.on('opentag', (tag) => {
switch (tag.name) {
case 'url':
case 'loc':
case 'urlset':
case 'lastmod':
case 'changefreq':
case 'priority':
case 'video:thumbnail_loc':
case 'video:video':
case 'video:title':
case 'video:description':
case 'video:tag':
case 'video:duration':
case 'video:player_loc':
case 'image:image':
case 'image:loc':
case 'image:geo_location':
case 'image:license':
case 'image:title':
case 'image:caption':
case 'video:requires_subscription':
case 'video:publication_date':
case 'video:id':
case 'video:restriction':
case 'video:family_friendly':
case 'video:view_count':
case 'video:uploader':
case 'video:expiration_date':
case 'video:platform':
case 'video:price':
case 'video:rating':
case 'video:category':
case 'video:live':
case 'video:gallery_loc':
case 'news:news':
case 'news:publication':
case 'news:name':
case 'news:access':
case 'news:genres':
case 'news:publication_date':
case 'news:title':
case 'news:keywords':
case 'news:stock_tickers':
case 'news:language':
case 'mobile:mobile':
break;
case 'xhtml:link':
if (isValidTagName(tag.name)) {
if (tag.name === 'xhtml:link') {
if (typeof tag.attributes.rel === 'string' ||
typeof tag.attributes.href === 'string') {
break;
return;
}

@@ -131,3 +90,3 @@ if (tag.attributes.rel.value === 'alternate' &&

if (typeof tag.attributes.hreflang === 'string')
break;
return;
currentLink.lang = tag.attributes.hreflang.value;

@@ -146,7 +105,7 @@ }

}
break;
default:
console.warn('unhandled tag', tag.name);
break;
}
}
else {
console.warn('unhandled tag', tag.name);
}
});

@@ -157,86 +116,86 @@ this.saxStream.on('text', (text) => {

break;
case 'loc':
case types_1.TagNames.loc:
currentItem.url = text;
break;
case 'changefreq':
currentItem.changefreq = text;
case types_1.TagNames.changefreq:
if (types_1.isValidChangeFreq(text)) {
currentItem.changefreq = text;
}
break;
case 'priority':
case types_1.TagNames.priority:
currentItem.priority = parseFloat(text);
break;
case 'lastmod':
case types_1.TagNames.lastmod:
currentItem.lastmod = text;
break;
case 'video:thumbnail_loc':
// eslint-disable-next-line @typescript-eslint/camelcase
case types_1.TagNames['video:thumbnail_loc']:
currentVideo.thumbnail_loc = text;
break;
case 'video:tag':
case types_1.TagNames['video:tag']:
currentVideo.tag.push(text);
break;
case 'video:duration':
case types_1.TagNames['video:duration']:
currentVideo.duration = parseInt(text, 10);
break;
case 'video:player_loc':
// eslint-disable-next-line @typescript-eslint/camelcase
case types_1.TagNames['video:player_loc']:
currentVideo.player_loc = text;
break;
case 'video:requires_subscription':
// eslint-disable-next-line @typescript-eslint/camelcase
currentVideo.requires_subscription = text;
case types_1.TagNames['video:requires_subscription']:
if (types_1.isValidYesNo(text)) {
currentVideo.requires_subscription = text;
}
break;
case 'video:publication_date':
// eslint-disable-next-line @typescript-eslint/camelcase
case types_1.TagNames['video:publication_date']:
currentVideo.publication_date = text;
break;
case 'video:id':
case types_1.TagNames['video:id']:
currentVideo.id = text;
break;
case 'video:restriction':
case types_1.TagNames['video:restriction']:
currentVideo.restriction = text;
break;
case 'video:view_count':
// eslint-disable-next-line @typescript-eslint/camelcase
currentVideo.view_count = text;
case types_1.TagNames['video:view_count']:
currentVideo.view_count = parseInt(text, 10);
break;
case 'video:uploader':
case types_1.TagNames['video:uploader']:
currentVideo.uploader = text;
break;
case 'video:family_friendly':
// eslint-disable-next-line @typescript-eslint/camelcase
currentVideo.family_friendly = text;
case types_1.TagNames['video:family_friendly']:
if (types_1.isValidYesNo(text)) {
currentVideo.family_friendly = text;
}
break;
case 'video:expiration_date':
// eslint-disable-next-line @typescript-eslint/camelcase
case types_1.TagNames['video:expiration_date']:
currentVideo.expiration_date = text;
break;
case 'video:platform':
case types_1.TagNames['video:platform']:
currentVideo.platform = text;
break;
case 'video:price':
case types_1.TagNames['video:price']:
currentVideo.price = text;
break;
case 'video:rating':
case types_1.TagNames['video:rating']:
currentVideo.rating = parseFloat(text);
break;
case 'video:category':
case types_1.TagNames['video:category']:
currentVideo.category = text;
break;
case 'video:live':
currentVideo.live = text;
case types_1.TagNames['video:live']:
if (types_1.isValidYesNo(text)) {
currentVideo.live = text;
}
break;
case 'video:gallery_loc':
// eslint-disable-next-line @typescript-eslint/camelcase
case types_1.TagNames['video:gallery_loc']:
currentVideo.gallery_loc = text;
break;
case 'image:loc':
case types_1.TagNames['image:loc']:
currentImage.url = text;
break;
case 'image:geo_location':
case types_1.TagNames['image:geo_location']:
currentImage.geoLocation = text;
break;
case 'image:license':
case types_1.TagNames['image:license']:
currentImage.license = text;
break;
case 'news:access':
case types_1.TagNames['news:access']:
if (!currentItem.news) {

@@ -247,3 +206,3 @@ currentItem.news = newsTemplate();

break;
case 'news:genres':
case types_1.TagNames['news:genres']:
if (!currentItem.news) {

@@ -254,10 +213,9 @@ currentItem.news = newsTemplate();

break;
case 'news:publication_date':
case types_1.TagNames['news:publication_date']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
// eslint-disable-next-line @typescript-eslint/camelcase
currentItem.news.publication_date = text;
break;
case 'news:keywords':
case types_1.TagNames['news:keywords']:
if (!currentItem.news) {

@@ -268,10 +226,9 @@ currentItem.news = newsTemplate();

break;
case 'news:stock_tickers':
case types_1.TagNames['news:stock_tickers']:
if (!currentItem.news) {
currentItem.news = newsTemplate();
}
// eslint-disable-next-line @typescript-eslint/camelcase
currentItem.news.stock_tickers = text;
break;
case 'news:language':
case types_1.TagNames['news:language']:
if (!currentItem.news) {

@@ -282,9 +239,9 @@ currentItem.news = newsTemplate();

break;
case 'video:title':
case types_1.TagNames['video:title']:
currentVideo.title += text;
break;
case 'video:description':
case types_1.TagNames['video:description']:
currentVideo.description += text;
break;
case 'news:name':
case types_1.TagNames['news:name']:
if (!currentItem.news) {

@@ -295,3 +252,3 @@ currentItem.news = newsTemplate();

break;
case 'news:title':
case types_1.TagNames['news:title']:
if (!currentItem.news) {

@@ -302,3 +259,3 @@ currentItem.news = newsTemplate();

break;
case 'image:caption':
case types_1.TagNames['image:caption']:
if (!currentImage.caption) {

@@ -311,3 +268,3 @@ currentImage.caption = text;

break;
case 'image:title':
case types_1.TagNames['image:title']:
if (!currentImage.title) {

@@ -327,9 +284,9 @@ currentImage.title = text;

switch (currentTag) {
case 'video:title':
case types_1.TagNames['video:title']:
currentVideo.title += text;
break;
case 'video:description':
case types_1.TagNames['video:description']:
currentVideo.description += text;
break;
case 'news:name':
case types_1.TagNames['news:name']:
if (!currentItem.news) {

@@ -340,3 +297,3 @@ currentItem.news = newsTemplate();

break;
case 'news:title':
case types_1.TagNames['news:title']:
if (!currentItem.news) {

@@ -347,3 +304,3 @@ currentItem.news = newsTemplate();

break;
case 'image:caption':
case types_1.TagNames['image:caption']:
if (!currentImage.caption) {

@@ -356,3 +313,3 @@ currentImage.caption = text;

break;
case 'image:title':
case types_1.TagNames['image:title']:
if (!currentImage.title) {

@@ -372,8 +329,8 @@ currentImage.title = text;

switch (currentTag) {
case 'urlset':
case 'xhtml:link':
case 'video:id':
case types_1.TagNames['urlset']:
case types_1.TagNames['xhtml:link']:
case types_1.TagNames['video:id']:
break;
case 'video:restriction':
if (attr.name === 'relationship') {
case types_1.TagNames['video:restriction']:
if (attr.name === 'relationship' && types_1.isAllowDeny(attr.value)) {
currentVideo['restriction:relationship'] = attr.value;

@@ -385,4 +342,4 @@ }

break;
case 'video:price':
if (attr.name === 'type') {
case types_1.TagNames['video:price']:
if (attr.name === 'type' && types_1.isPriceType(attr.value)) {
currentVideo['price:type'] = attr.value;

@@ -393,3 +350,3 @@ }

}
else if (attr.name === 'resolution') {
else if (attr.name === 'resolution' && types_1.isResolution(attr.value)) {
currentVideo['price:resolution'] = attr.value;

@@ -401,3 +358,3 @@ }

break;
case 'video:player_loc':
case types_1.TagNames['video:player_loc']:
if (attr.name === 'autoplay') {

@@ -410,11 +367,11 @@ currentVideo['player_loc:autoplay'] = attr.value;

break;
case 'video:platform':
if (attr.name === 'relationship') {
case types_1.TagNames['video:platform']:
if (attr.name === 'relationship' && types_1.isAllowDeny(attr.value)) {
currentVideo['platform:relationship'] = attr.value;
}
else {
console.log('unhandled attr for video:platform', attr.name);
console.log('unhandled attr for video:platform', attr.name, attr.value);
}
break;
case 'video:gallery_loc':
case types_1.TagNames['video:gallery_loc']:
if (attr.name === 'title') {

@@ -433,15 +390,15 @@ currentVideo['gallery_loc:title'] = attr.value;

switch (tag) {
case 'url':
case types_1.TagNames.url:
this.push(currentItem);
currentItem = tagTemplate();
break;
case 'video:video':
case types_1.TagNames['video:video']:
currentItem.video.push(currentVideo);
currentVideo = videoTemplate();
break;
case 'image:image':
case types_1.TagNames['image:image']:
currentItem.img.push(currentImage);
currentImage = { ...imageTemplate };
break;
case 'xhtml:link':
case types_1.TagNames['xhtml:link']:
if (!dontpushCurrentLink) {

@@ -462,3 +419,3 @@ currentItem.links.push(currentLink);

}
exports.XMLToISitemapOptions = XMLToISitemapOptions;
exports.XMLToSitemapItemStream = XMLToSitemapItemStream;
/**

@@ -478,15 +435,12 @@ Read xml and resolve with the configuration that would produce it or reject with

@param {Readable} xml what to parse
@return {Promise<ISitemapOptions>} resolves with a valid config that can be
passed to createSitemap. Rejects with an Error object.
@return {Promise<SitemapItem[]>} resolves with list of sitemap items that can be fed into a SitemapStream. Rejects with an Error object.
*/
async function parseSitemap(xml) {
// eslint-disable-next-line @typescript-eslint/ban-ts-ignore
// @ts-ignore
const urls = [];
return new Promise((resolve, reject) => {
xml
.pipe(new XMLToISitemapOptions())
.pipe(new XMLToSitemapItemStream())
.on('data', (smi) => urls.push(smi))
.on('end', () => {
resolve({ urls });
resolve(urls);
})

@@ -493,0 +447,0 @@ .on('error', (error) => {

/// <reference types="node" />
import { ISitemapItemOptionsLoose, ErrorLevel } from './types';
import { Transform, TransformOptions, TransformCallback, Readable } from 'stream';
import { ISitemapOptions } from './sitemap';
export declare const preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\" xmlns:news=\"http://www.google.com/schemas/sitemap-news/0.9\" xmlns:xhtml=\"http://www.w3.org/1999/xhtml\" xmlns:image=\"http://www.google.com/schemas/sitemap-image/1.1\" xmlns:video=\"http://www.google.com/schemas/sitemap-video/1.1\">";
import { SitemapItemLoose, ErrorLevel } from './types';
export interface NSArgs {
news: boolean;
video: boolean;
xhtml: boolean;
image: boolean;
custom?: string[];
}
export declare const closetag = "</urlset>";
export interface ISitemapStreamOpts extends TransformOptions, Pick<ISitemapOptions, 'hostname' | 'level' | 'lastmodDateOnly'> {
export interface SitemapStreamOptions extends TransformOptions {
hostname?: string;
level?: ErrorLevel;
lastmodDateOnly?: boolean;
xmlns?: NSArgs;
errorHandler?: (error: Error, level: ErrorLevel) => void;
}
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*/
export declare class SitemapStream extends Transform {

@@ -13,5 +30,7 @@ hostname?: string;

hasHeadOutput: boolean;
xmlNS: NSArgs;
private smiStream;
lastmodDateOnly: boolean;
constructor(opts?: ISitemapStreamOpts);
_transform(item: ISitemapItemOptionsLoose, encoding: string, callback: TransformCallback): void;
constructor(opts?: SitemapStreamOptions);
_transform(item: SitemapItemLoose, encoding: string, callback: TransformCallback): void;
_flush(cb: TransformCallback): void;

@@ -18,0 +37,0 @@ }

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
const sitemap_item_1 = require("./sitemap-item");
const stream_1 = require("stream");
const types_1 = require("./types");
const stream_1 = require("stream");
const sitemap_1 = require("./sitemap");
exports.preamble = '<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">';
const utils_1 = require("./utils");
const sitemap_item_stream_1 = require("./sitemap-item-stream");
const preamble = '<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
const getURLSetNs = ({ news, video, image, xhtml, custom, }) => {
let ns = preamble;
if (news) {
ns += ' xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"';
}
if (xhtml) {
ns += ' xmlns:xhtml="http://www.w3.org/1999/xhtml"';
}
if (image) {
ns += ' xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"';
}
if (video) {
ns += ' xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"';
}
if (custom) {
ns += ' ' + custom.join(' ');
}
return ns + '>';
};
exports.closetag = '</urlset>';
const defaultStreamOpts = {};
const defaultXMLNS = {
news: true,
xhtml: true,
image: true,
video: true,
};
const defaultStreamOpts = {
xmlns: defaultXMLNS,
};
/**
* A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream)
* for turning a
* [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)
* of either [SitemapItemOptions](#sitemap-item-options) or url strings into a
* Sitemap. The readable stream it transforms **must** be in object mode.
*/
class SitemapStream extends stream_1.Transform {

@@ -17,3 +51,6 @@ constructor(opts = defaultStreamOpts) {

this.level = opts.level || types_1.ErrorLevel.WARN;
this.smiStream = new sitemap_item_stream_1.SitemapItemStream({ level: opts.level });
this.smiStream.on('data', data => this.push(data));
this.lastmodDateOnly = opts.lastmodDateOnly || false;
this.xmlNS = opts.xmlns || defaultXMLNS;
}

@@ -23,5 +60,5 @@ _transform(item, encoding, callback) {

this.hasHeadOutput = true;
this.push(exports.preamble);
this.push(getURLSetNs(this.xmlNS));
}
this.push(sitemap_item_1.SitemapItem.justItem(sitemap_1.Sitemap.normalizeURL(item, this.hostname, this.lastmodDateOnly), this.level));
this.smiStream.write(utils_1.validateSMIOptions(utils_1.normalizeURL(item, this.hostname, this.lastmodDateOnly), this.level));
callback();

@@ -28,0 +65,0 @@ }

/// <reference types="node" />
import { URL } from 'url';
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
export declare enum EnumChangefreq {

@@ -12,3 +19,9 @@ DAILY = "daily",

}
export declare const validators: {
[index: string]: RegExp;
};
export declare function isPriceType(pt: string | PriceType): pt is PriceType;
export declare function isResolution(res: string): res is Resolution;
export declare const CHANGEFREQ: EnumChangefreq[];
export declare function isValidChangeFreq(freq: string): freq is EnumChangefreq;
export declare enum EnumYesNo {

@@ -22,2 +35,3 @@ YES = "YES",

}
export declare function isValidYesNo(yn: string): yn is EnumYesNo;
export declare enum EnumAllowDeny {

@@ -27,43 +41,171 @@ ALLOW = "allow",

}
export declare type ICallback<E extends Error, T> = (err?: E, data?: T) => void;
export interface INewsItem {
export declare function isAllowDeny(ad: string): ad is EnumAllowDeny;
/**
* https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190
*/
export interface NewsItem {
access?: 'Registration' | 'Subscription';
publication: {
name: string;
/**
* The `<language>` is the language of your publication. Use an ISO 639
* language code (2 or 3 letters).
*/
language: string;
};
/**
* @example 'PressRelease, Blog'
*/
genres?: string;
/**
* Article publication date in W3C format, using either the "complete date" (YYYY-MM-DD) format or the "complete date
* plus hours, minutes, and seconds"
*/
publication_date: string;
/**
* The title of the news article
* @example 'Companies A, B in Merger Talks'
*/
title: string;
/**
* @example 'business, merger, acquisition'
*/
keywords?: string;
/**
* @example 'NASDAQ:A, NASDAQ:B'
*/
stock_tickers?: string;
}
export interface ISitemapImg {
/**
* Sitemap Image
* https://support.google.com/webmasters/answer/178636?hl=en&ref_topic=4581190
*/
export interface Img {
/**
* The URL of the image
* @example 'https://example.com/image.jpg'
*/
url: string;
/**
* The caption of the image
* @example 'Thanksgiving dinner'
*/
caption?: string;
/**
* The title of the image
* @example 'Star Wars EP IV'
*/
title?: string;
/**
* The geographic location of the image.
* @example 'Limerick, Ireland'
*/
geoLocation?: string;
/**
* A URL to the license of the image.
* @example 'https://example.com/license.txt'
*/
license?: string;
}
interface IVideoItemBase {
interface VideoItemBase {
/**
* A URL pointing to the video thumbnail image file
* @example "https://rtv3-img-roosterteeth.akamaized.net/store/0e841100-289b-4184-ae30-b6a16736960a.jpg/sm/thumb3.jpg"
*/
thumbnail_loc: string;
/**
* The title of the video
* @example '2018:E6 - GoldenEye: Source'
*/
title: string;
/**
* A description of the video. Maximum 2048 characters.
* @example 'We play gun game in GoldenEye: Source with a good friend of ours. His name is Gruchy. Dan Gruchy.'
*/
description: string;
/**
* A URL pointing to the actual video media file. Should be one of the supported formats. HTML is not a supported
* format. Flash is allowed, but no longer supported on most mobile platforms, and so may be indexed less well. Must
* not be the same as the `<loc>` URL.
* @example "http://streamserver.example.com/video123.mp4"
*/
content_loc?: string;
/**
* A URL pointing to a player for a specific video. Usually this is the information in the src element of an `<embed>`
* tag. Must not be the same as the `<loc>` URL
* @example "https://roosterteeth.com/embed/rouletsplay-2018-goldeneye-source"
*/
player_loc?: string;
/**
* A string the search engine can append as a query param to enable automatic
* playback. Equivilant to auto play attr on player_loc tag.
* @example 'ap=1'
*/
'player_loc:autoplay'?: string;
/**
* The length of the video in seconds
* @example 600
*/
duration?: number;
/**
* The date after which the video will no longer be available.
* @example "2012-07-16T19:20:30+08:00"
*/
expiration_date?: string;
view_count?: string | number;
/**
* The number of times the video has been viewed
*/
view_count?: number;
/**
* The date the video was first published, in W3C format.
* @example "2012-07-16T19:20:30+08:00"
*/
publication_date?: string;
/**
* A short description of the broad category that the video belongs to. This is a string no longer than 256 characters.
* @example Baking
*/
category?: string;
/**
* Whether to show or hide your video in search results from specific countries.
* @example "IE GB US CA"
*/
restriction?: string;
'restriction:relationship'?: string;
/**
* Whether the countries in restriction are allowed or denied
* @example 'deny'
*/
'restriction:relationship'?: EnumAllowDeny;
gallery_loc?: string;
'gallery_loc:title'?: string;
/**
* The price to download or view the video. Omit this tag for free videos.
* @example "1.99"
*/
price?: string;
'price:resolution'?: string;
/**
* Specifies the resolution of the purchased version. Supported values are hd and sd.
* @example "HD"
*/
'price:resolution'?: Resolution;
/**
* Specifies the currency in ISO4217 format.
* @example "USD"
*/
'price:currency'?: string;
'price:type'?: string;
/**
* Specifies the purchase option. Supported values are rend and own.
* @example "rent"
*/
'price:type'?: PriceType;
/**
* The video uploader's name. Only one <video:uploader> is allowed per video. String value, max 255 characters.
* @example "GrillyMcGrillerson"
*/
uploader?: string;
/**
* Whether to show or hide your video in search results on specified platform types. This is a list of space-delimited
* platform types. See <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190> for more detail
* @example "tv"
*/
platform?: string;

@@ -73,26 +215,67 @@ id?: string;

}
export interface IVideoItem extends IVideoItemBase {
export declare type PriceType = 'rent' | 'purchase' | 'RENT' | 'PURCHASE';
export declare type Resolution = 'HD' | 'hd' | 'sd' | 'SD';
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItem extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag: string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: number;
family_friendly?: EnumYesNo;
/**
* Indicates whether a subscription (either paid or free) is required to view
* the video. Allowed values are yes or no.
*/
requires_subscription?: EnumYesNo;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo;
}
export interface IVideoItemLoose extends IVideoItemBase {
/**
* Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>
*/
export interface VideoItemLoose extends VideoItemBase {
/**
* An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated
* with a video or piece of content.
* @example ['Baking']
*/
tag?: string | string[];
/**
* The rating of the video. Supported values are float numbers.
* @example 2.5
*/
rating?: string | number;
family_friendly?: EnumYesNo | boolean;
requires_subscription?: EnumYesNo | boolean;
/**
* Indicates whether the video is a live stream. Supported values are yes or no.
*/
live?: EnumYesNo | boolean;
}
export interface ILinkItem {
/**
* https://support.google.com/webmasters/answer/189077
*/
export interface LinkItem {
/**
* @example 'en'
*/
lang: string;
url: string;
}
export interface ISitemapIndexItemOptions {
export interface IndexItem {
url: string;
lastmod?: string;
lastmodISO?: string;
}
interface ISitemapItemOptionsBase {
interface SitemapItemBase {
lastmod?: string;

@@ -102,3 +285,3 @@ changefreq?: EnumChangefreq;

priority?: number;
news?: INewsItem;
news?: NewsItem;
expires?: string;

@@ -108,3 +291,2 @@ androidLink?: string;

url: string;
cdata?: boolean;
}

@@ -114,6 +296,6 @@ /**

*/
export interface SitemapItemOptions extends ISitemapItemOptionsBase {
img: ISitemapImg[];
video: IVideoItem[];
links: ILinkItem[];
export interface SitemapItem extends SitemapItemBase {
img: Img[];
video: VideoItem[];
links: LinkItem[];
}

@@ -123,6 +305,6 @@ /**

*/
export interface ISitemapItemOptionsLoose extends ISitemapItemOptionsBase {
video?: IVideoItemLoose | IVideoItemLoose[];
img?: string | ISitemapImg | (string | ISitemapImg)[];
links?: ILinkItem[];
export interface SitemapItemLoose extends SitemapItemBase {
video?: VideoItemLoose | VideoItemLoose[];
img?: string | Img | (string | Img)[];
links?: LinkItem[];
lastmodfile?: string | Buffer | URL;

@@ -136,6 +318,64 @@ lastmodISO?: string;

export declare enum ErrorLevel {
/**
* Validation will be skipped and nothing logged or thrown.
*/
SILENT = "silent",
/**
* If an invalid value is encountered, a console.warn will be called with details
*/
WARN = "warn",
/**
* An Error will be thrown on encountering invalid data.
*/
THROW = "throw"
}
export declare enum TagNames {
url = "url",
loc = "loc",
urlset = "urlset",
lastmod = "lastmod",
changefreq = "changefreq",
priority = "priority",
'video:thumbnail_loc' = "video:thumbnail_loc",
'video:video' = "video:video",
'video:title' = "video:title",
'video:description' = "video:description",
'video:tag' = "video:tag",
'video:duration' = "video:duration",
'video:player_loc' = "video:player_loc",
'video:content_loc' = "video:content_loc",
'image:image' = "image:image",
'image:loc' = "image:loc",
'image:geo_location' = "image:geo_location",
'image:license' = "image:license",
'image:title' = "image:title",
'image:caption' = "image:caption",
'video:requires_subscription' = "video:requires_subscription",
'video:publication_date' = "video:publication_date",
'video:id' = "video:id",
'video:restriction' = "video:restriction",
'video:family_friendly' = "video:family_friendly",
'video:view_count' = "video:view_count",
'video:uploader' = "video:uploader",
'video:expiration_date' = "video:expiration_date",
'video:platform' = "video:platform",
'video:price' = "video:price",
'video:rating' = "video:rating",
'video:category' = "video:category",
'video:live' = "video:live",
'video:gallery_loc' = "video:gallery_loc",
'news:news' = "news:news",
'news:publication' = "news:publication",
'news:name' = "news:name",
'news:access' = "news:access",
'news:genres' = "news:genres",
'news:publication_date' = "news:publication_date",
'news:title' = "news:title",
'news:keywords' = "news:keywords",
'news:stock_tickers' = "news:stock_tickers",
'news:language' = "news:language",
'mobile:mobile' = "mobile:mobile",
'xhtml:link' = "xhtml:link",
'expires' = "expires"
}
export {};
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
// can't be const enum if we use babel to compile
// https://github.com/babel/babel/issues/8741
/**
* How frequently the page is likely to change. This value provides general
* information to search engines and may not correlate exactly to how often they crawl the page. Please note that the
* value of this tag is considered a hint and not a command. See
* <https://www.sitemaps.org/protocol.html#xmlTagDefinitions> for the acceptable
* values
*/
var EnumChangefreq;

@@ -15,11 +20,29 @@ (function (EnumChangefreq) {

})(EnumChangefreq = exports.EnumChangefreq || (exports.EnumChangefreq = {}));
exports.CHANGEFREQ = [
EnumChangefreq.ALWAYS,
EnumChangefreq.HOURLY,
EnumChangefreq.DAILY,
EnumChangefreq.WEEKLY,
EnumChangefreq.MONTHLY,
EnumChangefreq.YEARLY,
EnumChangefreq.NEVER,
];
const allowDeny = /^(?:allow|deny)$/;
exports.validators = {
'price:currency': /^[A-Z]{3}$/,
'price:type': /^(?:rent|purchase|RENT|PURCHASE)$/,
'price:resolution': /^(?:HD|hd|sd|SD)$/,
'platform:relationship': allowDeny,
'restriction:relationship': allowDeny,
restriction: /^([A-Z]{2}( +[A-Z]{2})*)?$/,
platform: /^((web|mobile|tv)( (web|mobile|tv))*)?$/,
language: /^zh-cn|zh-tw|([a-z]{2,3})$/,
genres: /^(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated)(, *(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated))*$/,
// eslint-disable-next-line @typescript-eslint/camelcase
stock_tickers: /^(\w+:\w+(, *\w+:\w+){0,4})?$/,
};
function isPriceType(pt) {
return exports.validators['price:type'].test(pt);
}
exports.isPriceType = isPriceType;
function isResolution(res) {
return exports.validators['price:resolution'].test(res);
}
exports.isResolution = isResolution;
exports.CHANGEFREQ = Object.values(EnumChangefreq);
function isValidChangeFreq(freq) {
return exports.CHANGEFREQ.includes(freq);
}
exports.isValidChangeFreq = isValidChangeFreq;
var EnumYesNo;

@@ -34,2 +57,6 @@ (function (EnumYesNo) {

})(EnumYesNo = exports.EnumYesNo || (exports.EnumYesNo = {}));
function isValidYesNo(yn) {
return /^YES|NO|[Yy]es|[Nn]o$/.test(yn);
}
exports.isValidYesNo = isValidYesNo;
var EnumAllowDeny;

@@ -40,2 +67,6 @@ (function (EnumAllowDeny) {

})(EnumAllowDeny = exports.EnumAllowDeny || (exports.EnumAllowDeny = {}));
function isAllowDeny(ad) {
return allowDeny.test(ad);
}
exports.isAllowDeny = isAllowDeny;
/**

@@ -46,5 +77,64 @@ * How to handle errors in passed in urls

(function (ErrorLevel) {
/**
* Validation will be skipped and nothing logged or thrown.
*/
ErrorLevel["SILENT"] = "silent";
/**
* If an invalid value is encountered, a console.warn will be called with details
*/
ErrorLevel["WARN"] = "warn";
/**
* An Error will be thrown on encountering invalid data.
*/
ErrorLevel["THROW"] = "throw";
})(ErrorLevel = exports.ErrorLevel || (exports.ErrorLevel = {}));
var TagNames;
(function (TagNames) {
TagNames["url"] = "url";
TagNames["loc"] = "loc";
TagNames["urlset"] = "urlset";
TagNames["lastmod"] = "lastmod";
TagNames["changefreq"] = "changefreq";
TagNames["priority"] = "priority";
TagNames["video:thumbnail_loc"] = "video:thumbnail_loc";
TagNames["video:video"] = "video:video";
TagNames["video:title"] = "video:title";
TagNames["video:description"] = "video:description";
TagNames["video:tag"] = "video:tag";
TagNames["video:duration"] = "video:duration";
TagNames["video:player_loc"] = "video:player_loc";
TagNames["video:content_loc"] = "video:content_loc";
TagNames["image:image"] = "image:image";
TagNames["image:loc"] = "image:loc";
TagNames["image:geo_location"] = "image:geo_location";
TagNames["image:license"] = "image:license";
TagNames["image:title"] = "image:title";
TagNames["image:caption"] = "image:caption";
TagNames["video:requires_subscription"] = "video:requires_subscription";
TagNames["video:publication_date"] = "video:publication_date";
TagNames["video:id"] = "video:id";
TagNames["video:restriction"] = "video:restriction";
TagNames["video:family_friendly"] = "video:family_friendly";
TagNames["video:view_count"] = "video:view_count";
TagNames["video:uploader"] = "video:uploader";
TagNames["video:expiration_date"] = "video:expiration_date";
TagNames["video:platform"] = "video:platform";
TagNames["video:price"] = "video:price";
TagNames["video:rating"] = "video:rating";
TagNames["video:category"] = "video:category";
TagNames["video:live"] = "video:live";
TagNames["video:gallery_loc"] = "video:gallery_loc";
TagNames["news:news"] = "news:news";
TagNames["news:publication"] = "news:publication";
TagNames["news:name"] = "news:name";
TagNames["news:access"] = "news:access";
TagNames["news:genres"] = "news:genres";
TagNames["news:publication_date"] = "news:publication_date";
TagNames["news:title"] = "news:title";
TagNames["news:keywords"] = "news:keywords";
TagNames["news:stock_tickers"] = "news:stock_tickers";
TagNames["news:language"] = "news:language";
TagNames["mobile:mobile"] = "mobile:mobile";
TagNames["xhtml:link"] = "xhtml:link";
TagNames["expires"] = "expires";
})(TagNames = exports.TagNames || (exports.TagNames = {}));

@@ -1,11 +0,13 @@

/*!
* Sitemap
* Copyright(c) 2011 Eugene Kalinin
* MIT Licensed
*/
/// <reference types="node" />
import { SitemapItemOptions, ErrorLevel } from './types';
import { Readable, ReadableOptions } from 'stream';
export declare function validateSMIOptions(conf: SitemapItemOptions, level?: ErrorLevel): SitemapItemOptions;
import { SitemapItem, ErrorLevel, SitemapItemLoose } from './types';
declare function handleError(error: Error, level: ErrorLevel): void;
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
export declare function validateSMIOptions(conf: SitemapItem, level?: ErrorLevel, errorHandler?: typeof handleError): SitemapItem;
/**
* Combines multiple streams into one

@@ -15,3 +17,3 @@ * @param streams the streams to combine

export declare function mergeStreams(streams: Readable[]): Readable;
export interface IReadLineStreamOptions extends ReadableOptions {
export interface ReadlineStreamOptions extends ReadableOptions {
input: Readable;

@@ -22,5 +24,5 @@ }

*/
export declare class ReadLineStream extends Readable {
export declare class ReadlineStream extends Readable {
private _source;
constructor(options: IReadLineStreamOptions);
constructor(options: ReadlineStreamOptions);
_read(size: number): void;

@@ -32,3 +34,2 @@ }

* @param stream a stream of line separated urls.
* @param opts
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess

@@ -52,1 +53,9 @@ */

export declare function chunk(array: any[], size?: number): any[];
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
export declare function normalizeURL(elem: string | SitemapItemLoose, hostname?: string, lastmodDateOnly?: boolean): SitemapItem;
export {};
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
/*!

@@ -7,21 +8,9 @@ * Sitemap

*/
Object.defineProperty(exports, "__esModule", { value: true });
const fs_1 = require("fs");
const stream_1 = require("stream");
const readline_1 = require("readline");
const url_1 = require("url");
const types_1 = require("./types");
const errors_1 = require("./errors");
const stream_1 = require("stream");
const readline_1 = require("readline");
const allowDeny = /^allow|deny$/;
const validators = {
'price:currency': /^[A-Z]{3}$/,
'price:type': /^rent|purchase|RENT|PURCHASE$/,
'price:resolution': /^HD|hd|sd|SD$/,
'platform:relationship': allowDeny,
'restriction:relationship': allowDeny,
restriction: /^([A-Z]{2}( +[A-Z]{2})*)?$/,
platform: /^((web|mobile|tv)( (web|mobile|tv))*)?$/,
language: /^zh-cn|zh-tw|([a-z]{2,3})$/,
genres: /^(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated)(, *(PressRelease|Satire|Blog|OpEd|Opinion|UserGenerated))*$/,
// eslint-disable-next-line @typescript-eslint/camelcase
stock_tickers: /^(\w+:\w+(, *\w+:\w+){0,4})?$/,
};
const types_2 = require("./types");
function validate(subject, name, url, level) {

@@ -32,5 +21,5 @@ Object.keys(subject).forEach((key) => {

const val = subject[key];
if (validators[key] && !validators[key].test(val)) {
if (types_2.validators[key] && !types_2.validators[key].test(val)) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidAttrValue(key, val, validators[key]);
throw new errors_1.InvalidAttrValue(key, val, types_2.validators[key]);
}

@@ -43,3 +32,17 @@ else {

}
function validateSMIOptions(conf, level = types_1.ErrorLevel.WARN) {
function handleError(error, level) {
if (level === types_1.ErrorLevel.THROW) {
throw error;
}
else if (level === types_1.ErrorLevel.WARN) {
console.warn(error.name, error.message);
}
}
/**
* Verifies all data passed in will comply with sitemap spec.
* @param conf Options to validate
* @param level logging level
* @param errorHandler error handling func
*/
function validateSMIOptions(conf, level = types_1.ErrorLevel.WARN, errorHandler = handleError) {
if (!conf) {

@@ -53,17 +56,7 @@ throw new errors_1.NoConfigError();

if (!url) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.NoURLError();
}
else {
console.warn('URL is required');
}
errorHandler(new errors_1.NoURLError(), level);
}
if (changefreq) {
if (types_1.CHANGEFREQ.indexOf(changefreq) === -1) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.ChangeFreqInvalidError();
}
else {
console.warn(`${url}: changefreq ${changefreq} is not valid`);
}
if (!types_1.isValidChangeFreq(changefreq)) {
errorHandler(new errors_1.ChangeFreqInvalidError(url, changefreq), level);
}

@@ -73,8 +66,3 @@ }

if (!(priority >= 0.0 && priority <= 1.0)) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.PriorityInvalidError();
}
else {
console.warn(`${url}: priority ${priority} is not valid`);
}
errorHandler(new errors_1.PriorityInvalidError(url, priority), level);
}

@@ -86,8 +74,3 @@ }

news.access !== 'Subscription') {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidNewsAccessValue();
}
else {
console.warn(`${url}: news access ${news.access} is invalid`);
}
errorHandler(new errors_1.InvalidNewsAccessValue(url, news.access), level);
}

@@ -99,8 +82,3 @@ if (!news.publication ||

!news.title) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidNewsFormat();
}
else {
console.warn(`${url}: missing required news property`);
}
errorHandler(new errors_1.InvalidNewsFormat(url), level);
}

@@ -112,19 +90,10 @@ validate(news, 'news', url, level);

video.forEach((vid) => {
var _a;
if (vid.duration !== undefined) {
if (vid.duration < 0 || vid.duration > 28800) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidVideoDuration();
}
else {
console.warn(`${url}: video duration ${vid.duration} is invalid`);
}
errorHandler(new errors_1.InvalidVideoDuration(url, vid.duration), level);
}
}
if (vid.rating !== undefined && (vid.rating < 0 || vid.rating > 5)) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidVideoRating();
}
else {
console.warn(`${url}: video ${vid.title} rating ${vid.rating} must be between 0 and 5 inclusive`);
}
errorHandler(new errors_1.InvalidVideoRating(url, vid.title, vid.rating), level);
}

@@ -136,17 +105,45 @@ if (typeof vid !== 'object' ||

// has to be an object and include required categories https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidVideoFormat();
}
else {
console.warn(`${url}: missing required video property`);
}
errorHandler(new errors_1.InvalidVideoFormat(url), level);
}
if (vid.title.length > 100) {
errorHandler(new errors_1.InvalidVideoTitle(url, vid.title.length), level);
}
if (vid.description.length > 2048) {
if (level === types_1.ErrorLevel.THROW) {
throw new errors_1.InvalidVideoDescription();
errorHandler(new errors_1.InvalidVideoDescription(url, vid.description.length), level);
}
if (vid.view_count !== undefined && vid.view_count < 0) {
errorHandler(new errors_1.InvalidVideoViewCount(url, vid.view_count), level);
}
if (vid.tag.length > 32) {
errorHandler(new errors_1.InvalidVideoTagCount(url, vid.tag.length), level);
}
if (vid.category !== undefined && ((_a = vid.category) === null || _a === void 0 ? void 0 : _a.length) > 256) {
errorHandler(new errors_1.InvalidVideoCategory(url, vid.category.length), level);
}
if (vid.family_friendly !== undefined &&
!types_1.isValidYesNo(vid.family_friendly)) {
errorHandler(new errors_1.InvalidVideoFamilyFriendly(url, vid.family_friendly), level);
}
if (vid.restriction) {
if (!types_2.validators.restriction.test(vid.restriction)) {
errorHandler(new errors_1.InvalidVideoRestriction(url, vid.restriction), level);
}
else {
console.warn(`${url}: video description is too long`);
if (!vid['restriction:relationship'] ||
!types_1.isAllowDeny(vid['restriction:relationship'])) {
errorHandler(new errors_1.InvalidVideoRestrictionRelationship(url, vid['restriction:relationship']), level);
}
}
// TODO price element should be unbounded
if ((vid.price === '' && vid['price:type'] === undefined) ||
(vid['price:type'] !== undefined && !types_1.isPriceType(vid['price:type']))) {
errorHandler(new errors_1.InvalidVideoPriceType(url, vid['price:type'], vid.price), level);
}
if (vid['price:resolution'] !== undefined &&
!types_1.isResolution(vid['price:resolution'])) {
errorHandler(new errors_1.InvalidVideoResolution(url, vid['price:resolution']), level);
}
if (vid['price:currency'] !== undefined &&
!types_2.validators['price:currency'].test(vid['price:currency'])) {
errorHandler(new errors_1.InvalidVideoPriceCurrency(url, vid['price:currency']), level);
}
validate(vid, 'video', url, level);

@@ -175,3 +172,3 @@ });

*/
class ReadLineStream extends stream_1.Readable {
class ReadlineStream extends stream_1.Readable {
constructor(options) {

@@ -205,3 +202,3 @@ if (options.autoDestroy === undefined) {

}
exports.ReadLineStream = ReadLineStream;
exports.ReadlineStream = ReadlineStream;
/**

@@ -211,7 +208,6 @@ * Takes a stream likely from fs.createReadStream('./path') and returns a stream

* @param stream a stream of line separated urls.
* @param opts
* @param opts.isJSON is the stream line separated JSON. leave undefined to guess
*/
function lineSeparatedURLsToSitemapOptions(stream, { isJSON } = {}) {
return new ReadLineStream({ input: stream }).pipe(new stream_1.Transform({
return new ReadlineStream({ input: stream }).pipe(new stream_1.Transform({
objectMode: true,

@@ -256,1 +252,117 @@ transform: (line, encoding, cb) => {

exports.chunk = chunk;
function boolToYESNO(bool) {
if (bool === undefined) {
return bool;
}
if (typeof bool === 'boolean') {
return bool ? types_1.EnumYesNo.yes : types_1.EnumYesNo.no;
}
return bool;
}
/**
* Converts the passed in sitemap entry into one capable of being consumed by SitemapItem
* @param {string | SitemapItemLoose} elem the string or object to be converted
* @param {string} hostname
* @returns SitemapItemOptions a strict sitemap item option
*/
function normalizeURL(elem, hostname, lastmodDateOnly = false) {
// SitemapItem
// create object with url property
let smi = {
img: [],
video: [],
links: [],
url: '',
};
let smiLoose;
if (typeof elem === 'string') {
smi.url = elem;
smiLoose = { url: elem };
}
else {
smiLoose = elem;
}
smi.url = new url_1.URL(smiLoose.url, hostname).toString();
let img = [];
if (smiLoose.img) {
if (typeof smiLoose.img === 'string') {
// string -> array of objects
smiLoose.img = [{ url: smiLoose.img }];
}
else if (!Array.isArray(smiLoose.img)) {
// object -> array of objects
smiLoose.img = [smiLoose.img];
}
img = smiLoose.img.map((el) => (typeof el === 'string' ? { url: el } : el));
}
// prepend hostname to all image urls
smi.img = img.map((el) => ({
...el,
url: new url_1.URL(el.url, hostname).toString(),
}));
let links = [];
if (smiLoose.links) {
links = smiLoose.links;
}
smi.links = links.map((link) => {
return { ...link, url: new url_1.URL(link.url, hostname).toString() };
});
if (smiLoose.video) {
if (!Array.isArray(smiLoose.video)) {
// make it an array
smiLoose.video = [smiLoose.video];
}
smi.video = smiLoose.video.map((video) => {
const nv = {
...video,
/* eslint-disable-next-line @typescript-eslint/camelcase */
family_friendly: boolToYESNO(video.family_friendly),
live: boolToYESNO(video.live),
/* eslint-disable-next-line @typescript-eslint/camelcase */
requires_subscription: boolToYESNO(video.requires_subscription),
tag: [],
rating: undefined,
};
if (video.tag !== undefined) {
nv.tag = !Array.isArray(video.tag) ? [video.tag] : video.tag;
}
if (video.rating !== undefined) {
if (typeof video.rating === 'string') {
nv.rating = parseFloat(video.rating);
}
else {
nv.rating = video.rating;
}
}
if (typeof video.view_count === 'string') {
/* eslint-disable-next-line @typescript-eslint/camelcase */
nv.view_count = parseInt(video.view_count, 10);
}
else if (typeof video.view_count === 'number') {
/* eslint-disable-next-line @typescript-eslint/camelcase */
nv.view_count = video.view_count;
}
return nv;
});
}
// If given a file to use for last modified date
if (smiLoose.lastmodfile) {
const { mtime } = fs_1.statSync(smiLoose.lastmodfile);
smi.lastmod = new Date(mtime).toISOString();
// The date of last modification (YYYY-MM-DD)
}
else if (smiLoose.lastmodISO) {
smi.lastmod = new Date(smiLoose.lastmodISO).toISOString();
}
else if (smiLoose.lastmod) {
smi.lastmod = new Date(smiLoose.lastmod).toISOString();
}
if (lastmodDateOnly && smi.lastmod) {
smi.lastmod = smi.lastmod.slice(0, 10);
}
delete smiLoose.lastmodfile;
delete smiLoose.lastmodISO;
smi = { ...smiLoose, ...smi };
return smi;
}
exports.normalizeURL = normalizeURL;
/// <reference types="node" />
import { Readable } from 'stream';
/**
* Verify the passed in xml is valid
* Verify the passed in xml is valid. Requires xmllib be installed
* @param xml what you want validated

@@ -6,0 +6,0 @@ * @return {Promise<null>} resolves on valid rejects [error stderr]

@@ -7,3 +7,3 @@ "use strict";

/**
* Verify the passed in xml is valid
* Verify the passed in xml is valid. Requires xmllib be installed
* @param xml what you want validated

@@ -10,0 +10,0 @@ * @return {Promise<null>} resolves on valid rejects [error stderr]

{
"name": "sitemap",
"version": "5.1.0",
"version": "6.0.0",
"description": "Sitemap-generating lib/cli",

@@ -42,6 +42,14 @@ "keywords": [

"hooks": {
"pre-commit": "sort-package-json",
"pre-push": "npm run test:fast"
"pre-commit": "lint-staged && npm run test:fast"
}
},
"lint-staged": {
"package.json": [
"sort-package-json"
],
"{lib,tests}/**/*.ts": [
"eslint --fix",
"prettier --write"
]
},
"eslintConfig": {

@@ -53,9 +61,2 @@ "env": {

},
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended",
"prettier/@typescript-eslint",
"plugin:prettier/recommended"
],
"parser": "@typescript-eslint/parser",

@@ -70,8 +71,11 @@ "parserOptions": {

],
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended",
"prettier/@typescript-eslint",
"plugin:prettier/recommended"
],
"rules": {
"no-case-declarations": 0,
"no-console": 0,
"no-unused-vars": 0,
"indent": "off",
"no-dupe-class-members": "off",
"lines-between-class-members": [

@@ -84,2 +88,6 @@ "error",

],
"no-case-declarations": 0,
"no-console": 0,
"no-dupe-class-members": "off",
"no-unused-vars": 0,
"padding-line-between-statements": [

@@ -93,2 +101,4 @@ "error",

],
"@typescript-eslint/explicit-member-accessibility": "off",
"@typescript-eslint/interface-name-prefix": "off",
"@typescript-eslint/no-parameter-properties": "off",

@@ -100,9 +110,19 @@ "@typescript-eslint/no-unused-vars": [

}
],
"@typescript-eslint/explicit-member-accessibility": "off",
"@typescript-eslint/interface-name-prefix": [
2,
"always"
]
}
},
"overrides": [
{
"files": [
"*.js"
],
"rules": {
"@typescript-eslint/explicit-function-return-type": [
"off"
],
"@typescript-eslint/no-var-requires": [
"off"
]
}
}
]
},

@@ -127,34 +147,36 @@ "jest": {

"dependencies": {
"@types/node": "^12.12.3",
"@types/sax": "^1.2.0",
"arg": "^4.1.1",
"sax": "^1.2.4",
"xmlbuilder": "^13.0.2"
"@types/node": "^13.7.4",
"@types/sax": "^1.2.1",
"arg": "^4.1.3",
"sax": "^1.2.4"
},
"devDependencies": {
"@babel/core": "^7.6.4",
"@babel/plugin-proposal-class-properties": "^7.5.5",
"@babel/plugin-transform-typescript": "^7.6.3",
"@babel/preset-env": "^7.6.3",
"@babel/preset-typescript": "^7.6.0",
"@types/jest": "^24.0.21",
"@typescript-eslint/eslint-plugin": "^2.6.0",
"@typescript-eslint/parser": "^2.6.0",
"@babel/core": "^7.8.4",
"@babel/plugin-proposal-class-properties": "^7.8.3",
"@babel/plugin-proposal-nullish-coalescing-operator": "^7.8.3",
"@babel/plugin-proposal-optional-chaining": "^7.8.3",
"@babel/plugin-transform-typescript": "^7.8.3",
"@babel/preset-env": "^7.8.4",
"@babel/preset-typescript": "^7.8.3",
"@types/jest": "^25.1.3",
"@typescript-eslint/eslint-plugin": "^2.20.0",
"@typescript-eslint/parser": "^2.20.0",
"babel-eslint": "^10.0.3",
"babel-polyfill": "^6.26.0",
"concurrently": "^5.0.0",
"eslint": "^6.6.0",
"eslint-config-prettier": "^6.5.0",
"eslint-plugin-jest": "^22.21.0",
"eslint-plugin-prettier": "^3.1.1",
"concurrently": "^5.1.0",
"eslint": "^6.8.0",
"eslint-config-prettier": "^6.10.0",
"eslint-plugin-jest": "^23.8.0",
"eslint-plugin-prettier": "^3.1.2",
"express": "^4.17.1",
"husky": "^3.0.9",
"jest": "^24.9.0",
"prettier": "^1.18.2",
"sort-package-json": "^1.22.1",
"husky": "^4.2.3",
"jest": "^25.1.0",
"lint-staged": "^10.0.7",
"prettier": "^1.19.1",
"sort-package-json": "^1.40.0",
"source-map": "~0.7.3",
"stats-lite": "^2.2.0",
"stream-json": "^1.3.1",
"stream-json": "^1.3.3",
"through2-map": "^3.0.0",
"typescript": "^3.6.4"
"typescript": "^3.8.2"
},

@@ -161,0 +183,0 @@ "engines": {

@@ -1,5 +0,5 @@

# sitemap.js [![Build Status](https://travis-ci.org/ekalinin/sitemap.js.svg?branch=master)](https://travis-ci.org/ekalinin/sitemap.js)
# sitemap ![MIT License](https://img.shields.io/npm/l/sitemap)[![Build Status](https://travis-ci.org/ekalinin/sitemap.js.svg?branch=master)](https://travis-ci.org/ekalinin/sitemap.js)![Monthly Downloads](https://img.shields.io/npm/dm/sitemap)
**sitemap.js** is a high-level sitemap-generating library/CLI that
makes creating [sitemap XML](http://www.sitemaps.org/) files easy.
**sitemap** is a high-level streaming sitemap-generating library/CLI that
makes creating [sitemap XML](http://www.sitemaps.org/) files easy. [What is a sitemap?](https://support.google.com/webmasters/answer/156184?hl=en&ref_topic=4581190)

@@ -16,3 +16,3 @@ ## Maintainers

- [CLI](#cli)
- [Example of using sitemap.js with <a href="https://expressjs.com/">express</a>](#example-of-using-sitemapjs-with-express)
- [Example of using sitemap.js with](#example-of-using-sitemapjs-with-express) [express](https://expressjs.com/)
- [Stream writing a sitemap](#stream-writing-a-sitemap)

@@ -22,18 +22,20 @@ - [Example of most of the options you can use for sitemap](#example-of-most-of-the-options-you-can-use-for-sitemap)

- [Auto creating sitemap and index files from one large list](#auto-creating-sitemap-and-index-files-from-one-large-list)
- [More](#more)
- [API](#api)
- [Sitemap (deprecated)](#sitemap---deprecated)
- [buildSitemapIndex](#buildsitemapindex)
- [SitemapStream](#sitemapstream)
- [XMLToSitemapOptions](#XMLToSitemapOptions)
- [sitemapAndIndexStream](#sitemapandindexstream)
- [SitemapIndexStream](#SitemapIndexStream)
- [createSitemapsAndIndex](#createsitemapsandindex)
- [xmlLint](#xmllint)
- [parseSitemap](#parsesitemap)
- [SitemapStream](#sitemapstream)
- [XMLToISitemapOptions](#XMLToISitemapOptions)
- [lineSeparatedURLsToSitemapOptions](#lineseparatedurlstositemapoptions)
- [streamToPromise](#streamtopromise)
- [ObjectStreamToJSON](#objectstreamtojson)
- [SitemapItemStream](#SitemapItemStream)
- [Sitemap Item Options](#sitemap-item-options)
- [ISitemapImage](#isitemapimage)
- [IVideoItem](#ivideoitem)
- [ILinkItem](#ilinkitem)
- [INewsItem](#inewsitem)
- [SitemapImage](#sitemapimage)
- [VideoItem](#videoitem)
- [LinkItem](#linkitem)
- [NewsItem](#newsitem)
- [License](#license)

@@ -57,2 +59,8 @@

Or create an index and sitemaps at the same time.
```sh
npx sitemap --index --index-base-url https://example.com/path/to/sitemaps/ < listofurls.txt > sitemap-index.xml
```
Or validate an existing sitemap (requires libxml)

@@ -283,131 +291,128 @@

```js
const { createSitemapsAndIndex } = require('sitemap')
const smi = createSitemapsAndIndex({
hostname: 'http://www.sitemap.org',
sitemapName: 'sm-test',
sitemapSize: 1,
targetFolder: require('os').tmpdir(),
urls: ['http://ya.ru', 'http://ya2.ru']
})
```
const limit = 45000
const baseURL = 'https://example.com/subdir/'
const sms = new SitemapAndIndexStream({
limit, // defaults to 45k
getSitemapStream: (i) => {
const sm = new SitemapStream();
const path = `./sitemap-${i}.xml`;
## API
### Sitemap - __deprecated__
```js
const { Sitemap } = require('sitemap')
const sm = new Sitemap({
urls: [{ url: '/path' }],
hostname: 'http://example.com',
cacheTime: 0, // default
level: 'warn', // default warns if it encounters bad data
lastmodDateOnly: false // relevant for baidu
})
sm.toString() // returns the xml as a string
if (argv['--gzip']) {
sm.pipe(createGzip()).pipe(createWriteStream(path));
} else {
sm.pipe(createWriteStream(path));
}
return [new URL(path, baseURL).toString(), sm];
},
});
let oStream = lineSeparatedURLsToSitemapOptions(
pickStreamOrArg(argv)
).pipe(sms);
if (argv['--gzip']) {
oStream = oStream.pipe(createGzip());
}
oStream.pipe(process.stdout);
```
#### toString
```js
sm.toString(true)
```
## More
Converts the urls stored in an instance of Sitemap to a valid sitemap xml document as a string. Accepts a boolean as its first argument to designate on whether to pretty print. Defaults to false.
#### toXML
For more examples see the [examples directory](./examples/)
alias for toString
## API
#### toGzip
### SitemapStream
```js
sm.toGzip ((xmlGzippedBuffer) => console.log(xmlGzippedBuffer))
sm.toGzip()
```
A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream) for turning a [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams) of either [SitemapItemOptions](#sitemap-item-options) or url strings into a Sitemap. The readable stream it transforms **must** be in object mode.
Like toString, it builds the xmlDocument, then it runs gzip on the resulting string and returns it as a Buffer via callback or direct invocation
#### clearCache
```js
sm.clearCache()
```javascript
const { SitemapStream } = require('sitemap')
const sms = new SitemapStream({
hostname: 'https://example.com', // optional only necessary if your paths are relative
lastmodDateOnly: false // defaults to false, flip to true for baidu
xmlNS: { // XML namespaces to turn on - all by default
news: true,
xhtml: true,
image: true,
video: true,
// custom: ['xmlns:custom="https://example.com"']
}
})
const readable = // a readable stream of objects
readable.pipe(sms).pipe(process.stdout)
```
Cache will be emptied and will be bypassed until set again
#### isCacheValid
### XMLToSitemapOptions
```js
sm.isCacheValid()
```
Takes a stream of xml and transforms it into a stream of ISitemapOptions.
Use this to parse existing sitemaps into config options compatible with this library
Returns true if it has been less than cacheTimeout ms since cache was set
#### setCache
```javascript
const { createReadStream, createWriteStream } = require('fs');
const { XMLToISitemapOptions, ObjectStreamToJSON } = require('sitemap');
```js
sm.setCache('...xmlDoc')
createReadStream('./some/sitemap.xml')
// turn the xml into sitemap option item options
.pipe(new XMLToISitemapOptions())
// convert the object stream to JSON
.pipe(new ObjectStreamToJSON())
// write the library compatible options to disk
.pipe(createWriteStream('./sitemapOptions.json'))
```
Stores the passed in string on the instance to be used when toString is called within the configured cacheTimeout returns the passed in string unaltered
#### add
### SitemapIndexStream
```js
sm.add('/path', 'warn')
```
Writes a sitemap index when given a stream urls.
Adds the provided url to the sitemap instance
takes an optional parameter level for whether to print a console warning in the event of bad data 'warn' (default),
throw an exception 'throw', or quietly ignore bad data 'silent'
returns the number of locations currently in the sitemap instance
#### contains
```js
sm.contains('/path')
/**
* writes the following
* <?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/</loc>
</sitemap>
<sitemap>
<loc>https://example.com/2</loc>
</sitemap>
*/
const smis = new SitemapIndexStream({level: 'warn'})
smis.write({url: 'https://example.com/'})
smis.write({url: 'https://example.com/2'})
smis.pipe(writestream)
smis.end()
```
Returns true if path is already a part of the sitemap instance, false otherwise.
#### del
### sitemapAndIndexStream
```js
sm.del('/path')
```
Use this to take a stream which may go over the max of 50000 items and split it into an index and sitemaps.
SitemapAndIndexStream consumes a stream of urls and streams out index entries while writing individual urls to the streams you give it.
Provide it with a function which when provided with a index returns a url where the sitemap will ultimately be hosted and a stream to write the current sitemap to. This function will be called everytime the next item in the stream would exceed the provided limit.
Removes the provided url or url option from the sitemap instance
#### normalizeURL
```js
Sitemap.normalizeURL('/', 'http://example.com', false)
```
const sms = new SitemapAndIndexStream({
limit, // defaults to 45k
getSitemapStream: (i) => {
const sm = new SitemapStream();
const path = `./sitemap-${i}.xml`;
Static function that returns the stricter form of a options passed to SitemapItem. The third argument is whether to use date-only varient of lastmod. For baidu.
#### normalizeURLs
```js
Sitemap.normalizeURLs(['http://example.com', {url: '/'}], 'http://example.com', false)
if (argv['--gzip']) {
sm.pipe(createGzip()).pipe(createWriteStream(path));
} else {
sm.pipe(createWriteStream(path));
}
return [new URL(path, baseURL).toString(), sm];
},
});
let oStream = lineSeparatedURLsToSitemapOptions(
pickStreamOrArg(argv)
).pipe(sms);
if (argv['--gzip']) {
oStream = oStream.pipe(createGzip());
}
oStream.pipe(process.stdout);
```
Static function that takes an array of urls and returns a Map of their resolved url to the strict form of SitemapItemOptions
### buildSitemapIndex
Build a sitemap index file
```js
const { buildSitemapIndex } = require('sitemap')
const index = buildSitemapIndex({
urls: [{ url: 'http://example.com/sitemap-1.xml', lastmod: '2019-07-01' }, 'http://example.com/sitemap-2.xml'],
lastmod: '2019-07-29'
})
```
### createSitemapsAndIndex
Create several sitemaps and an index automatically from a list of urls
Create several sitemaps and an index automatically from a list of urls. __deprecated__

@@ -458,34 +463,2 @@ ```js

### SitemapStream
A [Transform](https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream) for turning a [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams) of either [SitemapItemOptions](#sitemap-item-options) or url strings into a Sitemap. The readable stream it transforms **must** be in object mode.
```javascript
const { SitemapStream } = require('sitemap')
const sms = new SitemapStream({
hostname: 'https://example.com', // optional only necessary if your paths are relative
lastmodDateOnly: false // defaults to false, flip to true for baidu
})
const readable = // a readable stream of objects
readable.pipe(sms).pipe(process.stdout)
```
### XMLToISitemapOptions
Takes a stream of xml and transforms it into a stream of ISitemapOptions.
Use this to parse existing sitemaps into config options compatible with this library
```javascript
const { createReadStream, createWriteStream } = require('fs');
const { XMLToISitemapOptions, ObjectStreamToJSON } = require('sitemap');
createReadStream('./some/sitemap.xml')
// turn the xml into sitemap option item options
.pipe(new XMLToISitemapOptions())
// convert the object stream to JSON
.pipe(new ObjectStreamToJSON())
// write the library compatible options to disk
.pipe(createWriteStream('./sitemapOptions.json'))
```
### lineSeparatedURLsToSitemapOptions

@@ -521,2 +494,15 @@

### SitemapItemStream
Takes a stream of SitemapItemOptions and spits out xml for each
```js
// writes <url><loc>https://example.com</loc><url><url><loc>https://example.com/2</loc><url>
const smis = new SitemapItemStream({level: 'warn'})
smis.pipe(writestream)
smis.write({url: 'https://example.com', img: [], video: [], links: []})
smis.write({url: 'https://example.com/2', img: [], video: [], links: []})
smis.end()
```
### Sitemap Item Options

@@ -537,3 +523,3 @@

### ISitemapImage
### SitemapImage

@@ -551,3 +537,3 @@ Sitemap image

### IVideoItem
### VideoItem

@@ -561,3 +547,3 @@ Sitemap video. <https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190>

|description|string|'We play gun game in GoldenEye: Source with a good friend of ours. His name is Gruchy. Dan Gruchy.'|A description of the video. Maximum 2048 characters. |
|content_loc|string - optional|`"http://streamserver.example.com/video123.mp4"`|A URL pointing to the actual video media file. Should be one of the supported formats.HTML is not a supported format. Flash is allowed, but no longer supported on most mobile platforms, and so may be indexed less well. Must not be the same as the `<loc>` URL.|
|content_loc|string - optional|`"http://streamserver.example.com/video123.mp4"`|A URL pointing to the actual video media file. Should be one of the supported formats. HTML is not a supported format. Flash is allowed, but no longer supported on most mobile platforms, and so may be indexed less well. Must not be the same as the `<loc>` URL.|
|player_loc|string - optional|`"https://roosterteeth.com/embed/rouletsplay-2018-goldeneye-source"`|A URL pointing to a player for a specific video. Usually this is the information in the src element of an `<embed>` tag. Must not be the same as the `<loc>` URL|

@@ -567,3 +553,3 @@ |'player_loc:autoplay'|string - optional|'ap=1'|a string the search engine can append as a query param to enable automatic playback|

|expiration_date| string - optional|"2012-07-16T19:20:30+08:00"|The date after which the video will no longer be available|
|view_count|string - optional|'21000000000'|The number of times the video has been viewed.|
|view_count|number - optional|'21000000000'|The number of times the video has been viewed.|
|publication_date| string - optional|"2018-04-27T17:00:00.000Z"|The date the video was first published, in W3C format.|

@@ -584,3 +570,3 @@ |category|string - optional|"Baking"|A short description of the broad category that the video belongs to. This is a string no longer than 256 characters.|

|tag|string[] - optional|['Baking']|An arbitrary string tag describing the video. Tags are generally very short descriptions of key concepts associated with a video or piece of content.|
|rating|number - optional|2.5|The rating of the video. Supported values are float numbers i|
|rating|number - optional|2.5|The rating of the video. Supported values are float numbers|
|family_friendly|string 'YES'\|'NO' - optional|'YES'||

@@ -599,3 +585,3 @@ |requires_subscription|string 'YES'\|'NO' - optional|'YES'|Indicates whether a subscription (either paid or free) is required to view the video. Allowed values are yes or no.|

### INewsItem
### NewsItem

@@ -609,3 +595,3 @@ <https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190>

|publication['name']| string|'The Example Times'|The `<name>` is the name of the news publication. It must exactly match the name as it appears on your articles on news.google.com, except for anything in parentheses.|
|publication['language']|string|'en'|he `<language>` is the language of your publication. Use an ISO 639 language code (2 or 3 letters).|
|publication['language']|string|'en'|The `<language>` is the language of your publication. Use an ISO 639 language code (2 or 3 letters).|
|genres|string - optional|'PressRelease, Blog'||

@@ -612,0 +598,0 @@ |publication_date|string|'2008-12-23'|Article publication date in W3C format, using either the "complete date" (YYYY-MM-DD) format or the "complete date plus hours, minutes, and seconds"|

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc