NewsCrawler - a library to crawl online newspaper articles
It's a package for importing cost and usage data from various cloud platforms and SaaS tools
Website crawling bot
Basic web crawler
A web crawler that uses graph traversal algorithms to crawl the web.
Python askfm crawler.
A pluggable web crawler that extracts firmographic company data (funding, employees, locations, executives, financials) into validated Pydantic models.
A web spider to crawl public github repositories
Async crawler and parsing service for data.gouv.fr
A crawler of booli.se - a Swedish housing platform.
Basic web crawler that automates website exploration and producing web resource trees.
No more pain with iframes ...
Disables robots.txt for indexing by webcrawlers like Google
A Django app to block the IP addresses which sends too many hits to your application
Load additional crawlers from files
Verify a claimed AI-crawler identity: check the client IP against the operator's published ranges or forward-confirmed reverse DNS. 15 operator lists, 3052 prefixes bundled, zero dependencies.
Crawl and parse stock historical data
A python library for extracting data from html table
Tradingview scraper tool
One read() call for AI agents, stitched from 41 web tools: cheapest first, every page checked, a receipt for every attempt
Crawling Korea bank transactions
Configurable crawler for web-scraping
File Crawler index files and search hard-coded credentials.
Plugin for apispec which helps reusing the documentation by using a decorated function stack.
PTT library
Crawler integration with INSPIRE-HEP.
A Scrapling-aligned stealth web scraping library: adaptive selectors, TLS-fingerprint HTTP, JS rendering, and a spider framework.
Excel spreadsheet crawler and table parser for data discovery, extraction, and querying.
Aspose.HTML for Python via .NET is a powerful API for Python that provides a headless browser functionality, allowing you to work with HTML documents in a variety of ways. With this API, you can easily create new HTML documents or open existing ones from different sources. Once you have the document, you can perform various manipulation operations, such as removing and replacing HTML nodes.
Collection Of Tools To Ease Selenium Operation
Generic Crawler SDK
Universal documentation crawler that converts HTML pages to Markdown with internal link correction
Windows-optimized RAG MCP Server with Playwright crawling, Link Graph trees, and local hybrid search.
my simple crawler
SQLi Crawler with JavaScript support.