Crawler
botasaurus
The All in One Web Scraping Framework
is-bot
Python package to detect bots/crawlers/spiders via user-agent
markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page
crawler-zz-api
A sample Crawler API
pycrawlers
A collection of Crawlers
quick-crawler
A toolkit for quickly performing crawler functions
qte-parts-crawler
A web crawler for QTE parts data
crawlerdetect-coreteam
CrawlerDetect is a Python class for detecting bots/crawlers/spiders via the user agent.
crawler-update-detector
a test package
html-table-extractor
A python library for extracting data from html table
stock-crawler-utils
Utils for stock-crawler project
amo-partner-crawler
SELENIUM CRAWLER FOR SCRAPING BILLING DATA FROM AMOCRM PARTNER CABINET
crawler-test
This is a web application that extracts images URLs from web pages.
citation-crawler
Asynchronous high-concurrency citation crawler, use with caution!
gogo-crawler
A small example package
vianu-fraudcrawler
Intelligent Market Monitoring
fiftyone-devicedetection-cloud
51Degrees Device Detection parses HTTP headers to return detailed hardware, operating system, browser, and crawler information for the devices used to access your website or service. This package retrieves device detection results by consuming the 51Degrees cloud service.
dblp-crawler
异步高并发dblp爬虫，慎用
python-multiprocess-crawler
Python BaseClass for easier multiprocess web-crawling
news-crawlers
An extensible python library to create web crawlers which alert users on news.
favorites-crawler
Crawl your personal favorite images, photo albums, comics from website. Support pixiv, yande.re for now.
spidy-web-crawler
Spidy is the simple, easy to use command line web crawler.
geodatacrawler
a crawler script to extract and author metadata of spatial datasets
basecrawler
This is the crawler libray
fiftyone-devicedetection
51Degrees Device Detection parses HTTP headers to return detailed hardware, operating system, browser, and crawler information for the devices used to access your website or service. This is an alternative to popular UAParser, DeviceAtlas, and WURFL packages.
deficrawler
Python package to query DeFi data from several The Graph subgraphs
datalad-crawler
DataLad extension package for crawling external web resources into an automated data distribution
news-please
news-please is an open source easy-to-use news extractor that just works.
webcrawler
Web Crawler
facebook-crawler
Facebook crawler package can help you crawl the posts on public fanspages and groups from Facebook.
pylab-crawler-sdk
SDK for https://crawler.pylab.co
pt-crawler
a group of crawlers for private tracker website
pyaiocrawler
Asynchronous web crawler built on asyncio
selenium-crawler-template
Boilerplate for developing crawler with Selenium.
vmapcrawler
Crawler for Vmap map
crawler-rocket
Help you to build web crawlers easily and quickly
mdcrawler
Mildom(https://www.mildom.com/) crawler written in Python.
pixiv-crawler
A tool to download pixiv pictures
videolab-youtube-crawler
Clark University, Package for YouTube crawler and cleaning data
python-jobcrawler
Job scraper for LinkedIn, Indeed, Glassdoor
th2-grpc-crawler-data-processor
th2_grpc_crawler_data_processor
bamboo-crawler
a hobby crawler
products-crawler
A scrapy project for crawl product pictures and information.
pystock-crawler
Crawl and parse stock historical data
noizze-crawler
A web page crawler which returns (title, og:image, og:description).
image-crawler-utils
A rather customizable image crawler structure, designed to download images with their information using multi-threading method. Besides, several wheels have been implemented to help better build a custom image crawler for yourself.