Web Scraping Framework based on py3 asyncio
CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.
Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters.
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders.
Python SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)
CrawlerKeeper 节点代理程序：节点注册、心跳上报、任务拉取与并行执行、部署同步与日志指标上报
Browser fingerprint datapoints collected by Apify
DEPRECATED - Tools and constants shared across Apify projects.
CrawlerKeeper 爬虫 SDK：日志上报、结果写入、任务参数解析、大文件分片上传、TLS 指纹请求、代理管理与并发线程池
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper
Workspace CLI for configuring jobseek-crawler companies
Scrapling is an undetectable, powerful, flexible, high-performance Python framework that makes Web Scraping easy and effortless as it should be!
Stealthy Crawling. Maximum Results. A pluggable anti-bot and stealth framework for Scrapy.
Python SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)
Command-line program to download image galleries and collections from several image hosting sites
Tools for Crawlers
muchu's utility module
Tools for Python
crawler commons
Crawlera middleware for Scrapy
Crawl the public data from Tefas.
Framework for crawling
Library to build content for User-Agent HTTP header
news-please is an open source easy-to-use news extractor that just works.
Tools for Spiders
This project is dedicated to sharing and teaching the fundamentals and techniques of web reverse engineering. Web reverse engineering involves analyzing how websites and web applications work, and cracking or modifying existing code to achieve specific objectives. This project includes a series of tutorials, practical tools, and case studies aimed at helping developers, security researchers, and enthusiasts understand how to effectively reverse engineer web technologies.
A fast web crawler to satisfy all your needs
Searchstax Simulator Crawler
AI-powered geographic information crawler with multi-model support
Helps retrieve values from a JSON.
Crawl, extract and push climate metadata for indexing.
A simple crawler for http://www.meitulu.com/t/changtui/.
A lightweight Python crawling framework.
A simple python script to identify and classify possible crawlers through analysis of web server log files
Declarative YAML workflow engine for browser automation and web scraping
A client implementation of Firefox DevTools over remote debug protocol.
Web crawler that builds a graph of a site. HTTP → TLS → browser escalation with pluggable browser engines.
A pluggable SDK for extracting author information from academic publisher websites
Python package to detect bots/crawlers/spiders via user-agent
A light weight crawler which gives search results in HTML form or in Dictionary form, given urls and keywords.
Convert web content to Markdown & JSON files to fuel your GPTs and agent AI!
The All in One Web Scraping Framework
A shared library for web scraping utilities.
Core programs for crawling