Crawler

Web Scraping Framework based on py3 asyncio

CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

crawler commons

Framework for crawling

Class that provides decorators and functions for easy handling of crawlera sessions in a scrapy spider.

Tools for Spiders

Core programs for crawling

A web Crawler for DTC(dans ton chat), VDM(vie de merde) and SCMB(se coucher moins bete)

An image crawler, including multiple modules and GUI.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.

Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters.

A multi-thread crawler framework with many builtin image crawlers provided.

采集工具

Automate downloads using predefined sites and the My JDownloader API

Python Test Crawler

Python implementation Bloom filter

A distributed crawler framework based on Python

Crawlera middleware for Scrapy

Proxy rotation with PostgreSQL

crawler_studio

爬虫工具

Open source tool to display/filter/export information about PCI or PCI Express devices, as well as their topology.

Clark University, Package for YouTube crawler and cleaning data

An app to download novels from online sources and generate e-books.

A client to interact with freud-net API

A sample Crawler API

Command-line program to download image galleries and collections from several image hosting sites

异步高并发dblp爬虫，慎用

Search sites for RSS, Atom, and JSON feeds

删除多余代码

A common package for crawling

Asynchronous high-concurrency citation crawler, use with caution!

A rather customizable image crawler structure, designed to download images with their information using multi-threading method. Besides, several wheels have been implemented to help better build a custom image crawler for yourself.

a group of crawlers for private tracker website

CrawlerDetect is a Python class for detecting bots/crawlers/spiders via the user agent.

A sample Crawler API

This is a web application that extracts images URLs from web pages.

Scrapy utils for Modis crawlers projects.

Apify SDK for Python

Python package to query DeFi data from several The Graph subgraphs

DataLad extension package for crawling external web resources into an automated data distribution

SELENIUM CRAWLER FOR SCRAPING BILLING DATA FROM AMOCRM PARTNER CABINET

template tools

Crawlee for Python

Crawl your personal favorite images, photo albums, comics from website. Support pixiv, yande.re for now.

this is an aparat crawler library