New Case Study:See how Anthropic automated 95% of dependency reviews with Socket.Learn More
Socket
Sign inDemoInstall
Socket

reppy

Package Overview
Dependencies
Maintainers
4
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

reppy

Replacement robots.txt Parser

0.4.14
PyPI
Maintainers
4

Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:

  • Memoization of fetched robots.txt
  • Expiration taken from the Expires header
  • Batch queries
  • Configurable user agent for fetching robots.txt
  • Automatic refetching based on expiration

FAQs

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts