Web Scraping Framework based on py3 asyncio
CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.
Filter, clean, and sample URLs for web crawling, with spam, content type, and language filters.
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders.
CrawlerKeeper 节点代理程序:节点注册、心跳上报、任务拉取与并行执行、部署同步与日志上报
Python SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)
Browser fingerprint datapoints collected by Apify
DEPRECATED - Tools and constants shared across Apify projects.
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper
Scrapling is an undetectable, powerful, flexible, high-performance Python framework that makes Web Scraping easy and effortless as it should be!
CrawlerKeeper 爬虫 SDK:日志上报、结果写入、任务参数解析、大文件分片上传、TLS 指纹请求、代理管理与并发线程池
Stealthy Crawling. Maximum Results. A pluggable anti-bot and stealth framework for Scrapy.
Python SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)
Workspace CLI for configuring jobseek-crawler companies
muchu's utility module
Command-line program to download image galleries and collections from several image hosting sites
crawler commons
Crawl 3GPP FTP server, cache all(!) 3GPP document data, return structured JSON/YAML for each document - all via CLI.
Tools for Crawlers
广东信息价 MCP:内置多市多月结构化材料信息价直查(支持按月份回溯/环比),另含联网搜索/抓网页/PDF兜底
Framework for crawling
Regex-based dataset crawler, indexer, and artifact toolkit
Tools for Python
Crawler4j module SDK and CLI for building standard crawler4j modules
Crawlera middleware for Scrapy
Library to build content for User-Agent HTTP header
A pluggable SDK for extracting author information from academic publisher websites
Declarative YAML workflow engine for browser automation and web scraping
Collect and standardize data on cloud server offerings, include specs, prices, and measured performance benchmarks.
Convert web content to Markdown & JSON files to fuel your GPTs and agent AI!
Agentic Crawler Discovery Framework.
Crawler detection from User-Agent strings in 50 ns. Zero deps, no regex, ReDoS-safe.
Python package to detect bots/crawlers/spiders via user-agent
The official Python library for the crawler.dev API
A shared library for web scraping utilities.
Crawl the public data from Tefas.
Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
A simple crawler for http://www.meitulu.com/t/changtui/.
A simple python script to identify and classify possible crawlers through analysis of web server log files
A web Crawler for DTC(dans ton chat), VDM(vie de merde) and SCMB(se coucher moins bete)
A client implementation of Firefox DevTools over remote debug protocol.
Embedded persistent knowledge graph: absorb a crawl, keep it fresh on an adaptive TTL, answer with URLs and anchors inside a token budget.
Core programs for crawling
news-please is an open source easy-to-use news extractor that just works.