Security News
tea.xyz Spam Plagues npm and RubyGems Package Registries
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
xml-sitemap-url-scraper
Advanced tools
Readme
Call the function with an array of one or more XML sitemap urls. All sitemap urls provided in this array must not be compressed. Compressed sitemaps are only supported when nested under <sitemapindex>
tags.
Normal: https://www.example.com/sitemap.xml
Compressed: https://www.example.com/sitemap.xml.gz
Returns a promise that resolves with an array of all URLs from those sitemaps
Install
npm install --save xml-sitemap-url-scraper
Requirements
To be compatible with async in example
Demo
npm start
Example
const { sitemapUrlScraper } = require("xml-sitemap-url-scraper");
let sitemapUrls = [
"https://www.example.com/sitemap.xml"
]
// Define how many compressed sitemaps we want to decompress and process at once (if any are found)
let concurrency = 5;
// Function's concurrency defaults to 1 if no param is provided
let urls = sitemapUrlScraper(sitemapUrls, concurrency);
urls.then(result => {
console.log("Returned URLs: ", result);
})
.catch(err => {
console.log(err);
})
Example Output
Returned URLs: [ 'https://www.example.com/page_one',
'https://www.example.com/page_two' ]
<sitemapindex>
tagsNested sitemaps will automatically be traversed, and their urls will be included in the final output array. This currently only supports 1 level of nesting.
Exmaple of nested sitemaps
<sitemapindex xmlns="https://www.example.com/schemas/sitemap/0.84">
<sitemap>
<loc>https://www.example.com/edu/sitemap.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.example.com/gmail/sitemap.xml</loc>
</sitemap>
</sitemapindex>
Child sitemaps that are nested inside <sitemapindex>
tags will be decompressed, and their urls will be included in the final output array. Compressed sitemaps are processed concurrently according to the parameter provided in the function call. This can help avoid memory and CPU load issues when processing a large number of compressed sitemaps. This function may take a long time to execute if there is a significant number of compressed sitemaps being scraped.
Example of compressed sitemaps
<sitemapindex xmlns="https://www.example.com/schemas/sitemap/0.84">
<sitemap>
<loc>https://www.example.com/edu/sitemap.xml.gz</loc>
</sitemap>
<sitemap>
<loc>https://www.example.com/gmail/sitemap.xml.gz</loc>
</sitemap>
</sitemapindex>
FAQs
Scrapes all urls from an xml sitemap, or an array of xml sitemaps, including nested xml sitemaps.
The npm package xml-sitemap-url-scraper receives a total of 193 weekly downloads. As such, xml-sitemap-url-scraper popularity was classified as not popular.
We found that xml-sitemap-url-scraper demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
Security News
As cyber threats become more autonomous, AI-powered defenses are crucial for businesses to stay ahead of attackers who can exploit software vulnerabilities at scale.
Security News
UnitedHealth Group disclosed that the ransomware attack on Change Healthcare compromised protected health information for millions in the U.S., with estimated costs to the company expected to reach $1 billion.