robotexclusionrulesparser
Advanced tools
Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL.
This module has some features that the standard library module robotparser does not, including the ability to decode non-ASCII robots.txt files, respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.
Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html.
Robotexclusionrulesparser is released under a BSD license.
FAQs
A robots.txt parser alternative to Python's robotparser module
We found that robotexclusionrulesparser demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Socket Fix 2.0 brings targeted CVE remediation, smarter upgrade planning, and broader ecosystem support to help developers get to zero alerts.
Security News
Socket CEO Feross Aboukhadijeh joins Risky Business Weekly to unpack recent npm phishing attacks, their limited impact, and the risks if attackers get smarter.
Product
Socket’s new Tier 1 Reachability filters out up to 80% of irrelevant CVEs, so security teams can focus on the vulnerabilities that matter.
