Browser-based Instagram research tooling: Selenium scraper with optional GCS and PostgreSQL.
A web crawler for GPTs to build knowledge bases
Scrapy utils for Modis crawlers projects.
A toolkit for quickly performing crawler functions
PTT crawler using asyncio
A library for following automata based programming model.
An integration package connecting TheCrawler and LangChain
The crawler module
多源金融资讯 MCP 服务
A lightweight Python crawling framework.
Offline AI-crawler user-agent classifier: is_ai_crawler(ua) and identify(ua) -> {name, operator, category, verification_method}. 150 crawlers bundled, zero dependencies, no network.
Used for crawling arxiv papers automatically.
An SEO tool that analyzes the structure of a site, crawls the site, count words in the body of the site and warns of any technical SEO issues.
Cryptocurrency exchange announcement news crawler for major crypto exchanges
Facebook crawler package can help you crawl the posts on public fanspages and groups from Facebook.
Advanced browser-activity capture, behavioral profiling, alerting and reporting library for Python.
A Python MCP (Model Context Protocol) client for connecting to remote web crawler servers.
Crawler for Schul-Cloud Ressources
A fully functioned Focused Web Crawler using Genetic Algorithm
Mildom(https://www.mildom.com/) crawler written in Python.
Python BaseClass for easier multiprocess web-crawling
A professional crawler for American Chemical Society papers with modern web dashboard
轻量、异步、开箱即用的社交媒体聚合解析库
A web page crawler which returns (title, og:image, og:description).
Crawl and download videos from the CDVL (Consumer Digital Video Library) research repository
FamilySearch Crawler - extracts files for RedBlackGraph ingestion
Google picture crawler with SQS connection
中化采购平台公告抓取 MCP 服务
Scraping and Crawling Micro-Framework
Crawler for Shopee from all country
Crawl sites, extract text and metadata, index it in Solr
A package to assist in crawling Autotrader results
Disables robots.txt for indexing by webcrawlers like Google
Autonomous Scraping Agent: Scrape URLs with prompts and schema
Web crawler for datatist.com
A short description of your package
Autonomous AI Search, Training & Agent Crawler Diagnostic Engine
Asynchronous high-concurrency citation crawler, use with caution!
Daum News Crawler MCP Server for Midterm Project
The aiocrawler is a asynchronous/distributed web crawler/spider
File system crawler for OpenSearch / Elasticsearch — Python rewrite of FSCrawler
Searchstax Simulator Crawler
Python Synonyms Crawler Implementation
A simple crawler for proxy server list
crawler oa(oi, acm) competition info
Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot...) can reach a list of sites: robots.txt, X-Robots-Tag, noai meta tags and the real HTTP answer to each bot's user agent.
ACO Web Crawler implemented by python