Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Retrieves all URLs of a given sitemap.xml URL and fetches each page one by one. Useful for (load) testing the entire site for error responses.
$ pip install fetch-sitemap
$ fetch-sitemap --help
Usage: fetch-sitemap [OPTIONS] SITEMAP_URL
Fetch a given sitemap and retrieve all URLs in it.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --basic-auth -a TEXT Basic auth information. Format: 'username:password' │
│ --limit -l INT [>=1] Maximum number of URLs to fetch from the given sitemap.xml. │
│ --recursive/--no-recursive Recursively fetch all sitemap documents from the given sitemap.xml. [default: recursive] │
│ --concurrency-limit -c INT [>=1] Max number of concurrent requests. [default: 5; >=1] │
│ --request-timeout -t INT [>=1] Timeout for fetching a URL in seconds. [default: 30; >=1] │
│ --random -r Append a random string like ?12334232343 to each URL to bypass frontend cache. │
│ --random-length INT [1 to 100] Length of the --random hash. [default: 15; 1 to 100] │
│ --report-path -p FILE Store results in a CSV file. Example: ./report.csv │
│ --output-dir -o DIRECTORY Store all fetched sitemap documents in this folder. Example: /tmp/my.domain.com/ │
│ --slow-threshold FLOAT [>=0.0] Responses slower than this (in seconds) are considered 'slow'. [default: 5.0; >=0.0] │
│ --slow-num INTEGER OR "ALL" How many 'slow' responses to show. [default: 10] │
│ --user-agent TEXT User-Agent string set in the HTTP header. [default: Mozilla/5.0 (compatible; fetch-sitemap/23)] │
│ --version Show the version and exit. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
poetry install
poetry run fetch-sitemap -h
poetry run ./tests.sh
FAQs
Fetch a given sitemap and retrieve all URLs in it.
We found that fetch-sitemap demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.