Security News
The Dark Side of Open Source
At Node Congress, Socket CEO Feross Aboukhadijeh uncovers the darker aspects of open source, where applications that rely heavily on third-party dependencies can be exploited in supply chain attacks.
sitemap-generator-cli
Advanced tools
Readme
Create xml sitemaps from the command line.
Generates a sitemap by crawling your site. Uses streams to efficiently write the sitemap to your drive. Is cappable of creating multiple sitemaps if threshold is reached. Respects robots.txt and meta tags.
This module is available on npm.
npm install -g sitemap-generator-cli
# or execute it directly with npx (since npm v5.2)
npx sitemap-generator-cli https://example.com
The crawler will fetch all folder URL pages and file types parsed by Google. If present the robots.txt
will be taken into account and possible rules are applied for each URL to consider if it should be added to the sitemap. Also the crawler will not fetch URL's from a page if the robots meta tag with the value nofollow
is present and ignore them completely if noindex
rule is present. The crawler is able to apply the base
value to found links.
sitemap-generator [options] <url>
When the crawler finished the XML Sitemap will be built and saved to your specified filepath. If the count of fetched pages is greater than 50000 it will be splitted into several sitemap files and create a sitemapindex file. Google does not allow more than 50000 items in one sitemap.
Example:
sitemap-generator http://example.com
sitemap-generator --help
Usage: cli [options] <url>
Options:
-V, --version output the version number
-f, --filepath <filepath> path to file including filename (default: sitemap.xml)
-m, --max-entries <maxEntries> limits the maximum number of URLs per sitemap file (default: 50000)
-d, --max-depth <maxDepth> limits the maximum distance from the original request (default: 0)
-q, --query consider query string
-u, --user-agent <agent> set custom User Agent
-v, --verbose print details when crawling
-c, --max-concurrency <maxConcurrency> maximum number of requests the crawler will run simultaneously (default: 5)
-r, --no-respect-robots-txt controls whether the crawler should respect rules in robots.txt
-l, --last-mod add Last-Modified header to xml
-g, --change-freq <changeFreq> adds a <changefreq> line to each URL in the sitemap.
-p, --priority-map <priorityMap> priority for each depth url, values between 1.0 and 0.0, example: "1.0,0.8 0.6,0.4"
-h, --help output usage information
Path to file to write including the filename itself. Path can be absolute or relative. Default is sitemap.xml
.
Examples:
sitemap.xml
mymap.xml
/var/www/sitemap.xml
./sitemap.myext
Sets the maximum number of requests the crawler will run simultaneously (default: 5).
Define a limit of URLs per sitemap files, useful for site with lots of urls. Defaults to 50000.
Set a maximum distance from the original request to crawl URLs, useful for generating smaller sitemap.xml
files. Defaults to 0, which means it will crawl all levels.
Controls whether the crawler should respect rules in robots.txt.
Consider URLs with query strings like http://www.example.com/?foo=bar
as individual sites and add them to the sitemap.
Set a custom User Agent used for crawling. Default is Node/SitemapGenerator
.
Print debug messages during crawling process. Also prints out a summery when finished.
add Last-Modified header to xml
adds a line to each URL in the sitemap.
add priority for each depth url, values between 1.0 and 0.0, example: "1.0,0.8 0.6,0.4"
FAQs
Create xml sitemaps from the command line.
The npm package sitemap-generator-cli receives a total of 664 weekly downloads. As such, sitemap-generator-cli popularity was classified as not popular.
We found that sitemap-generator-cli demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
At Node Congress, Socket CEO Feross Aboukhadijeh uncovers the darker aspects of open source, where applications that rely heavily on third-party dependencies can be exploited in supply chain attacks.
Research
Security News
The Socket Research team found this npm package includes code for collecting sensitive developer information, including your operating system username, Git username, and Git email.
Security News
OpenJS is warning of social engineering takeovers targeting open source projects after receiving a credible attempt on the foundation.