Crawler

th2_grpc_crawler_data_processor

A simple web-crawling framework, based on aiohttp.

File Crawler index files and search hard-coded credentials.

Simple, powerful web crawler

MtG deck legality checker

Extracts information from different systems and convert them to dbt exposures

Tools for working with AWS Organizations

Disables robots.txt for indexing by webcrawlers like Google

Utils for stock-crawler project

A simple and efficient web crawler in Python.

Disables robots.txt for indexing by webcrawlers like Google

A collection of tools for searching the internets.

A toolkit for quickly performing crawler functions

Python BaseClass for easier multiprocess web-crawling

Asynchronous web crawler built on asyncio

Spidy is the simple, easy to use command line web crawler.

Used for crawling arxiv papers automatically.

Search SMB shares for interesting files

a crawler to make custom filtering on frtichi.fr

Async crawler and parsing service for data.gouv.fr

a hobby crawler

SDK for https://crawler.pylab.co

Crawling the web made easy.

Boilerplate for developing crawler with Selenium.

Scrapy Crawler for RQ

A short description of your package

Crawler for Vmap map

A micro-framework to crawl the web pages with crawlers configs. It can use MongoDB, Elasticsearch and Solr databases to cache and save the extracted data.

A rock-solid cryprocurrency crawler

a simple joox web crawler

Scalable and Compliant Web Crawler

A scrapy project for crawl product pictures and information.

TRENDYOL CRAWLER

Disables robots.txt for indexing by webcrawlers like Google

Language Crawler

Autonomous Scraping Agent: Scrape URLs with prompts and schema

Basic web crawler that automates website exploration and producing web resource trees.

Pull and standardize data on cloud compute resources.

A distributed network crawler framework

A simple web scraper for downloading images, tables, and text from a webpage.

A mitmproxy addon for background passive analysis, crawling, and basic active scanning.

A package for crawling markdown formatted articles from certain webpage and storing them locally.

Clark University, Package for YouTube crawler and cleaning data

A python crawler authored by Ken.

A package to assist in crawling Autotrader results

A library for crawling websites

简易、强大的推特（Twitter）采集程序,支持元搜索,用户,粉丝,关注,发文,回复,评论等采集

A gevent-based simple instagram hashtag crawler

A modern web crawler framework for Python