Crawler

CrawlerDetect is a Python class for detecting bots/crawlers/spiders via the user agent.

A client to interact with freud-net API

This is a web application that extracts images URLs from web pages.

A new crawler for LinkAhead

news-please is an open source easy-to-use news extractor that just works.

Facebook crawler package can help you crawl the posts on public fanspages and groups from Facebook.

A distributed crawler for getting info about DNS domains and services attached to them.

A simple web-crawling framework, based on aiohttp.

A python library for extracting data from html table

A rather customizable image crawler structure, designed to download images with their information using multi-threading method. Besides, several wheels have been implemented to help better build a custom image crawler for yourself.

Disables robots.txt for indexing by webcrawlers like Google

A small example package

Video Crawler

Cryptocurrency exchange announcement news crawler for major crypto exchanges

Autonomous Scraping Agent: Scrape URLs with prompts and schema

This is the crawler libray

Crawler integration with INSPIRE-HEP.

A modular, async Python framework for structured online data collection used by Ollama Agent Roll Cage (OARC)

Help you to build web crawlers easily and quickly

a test package

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page

The All in One Web Scraping Framework

Python wrapper for crawler

Intelligent Market Monitoring

Intercept, decrypt and forward CnL to FeedCrawler

Web Crawler

a crawler script to extract and author metadata of spatial datasets

A distributed network crawler framework

Crawl telegra.ph searching for nudes!

A toolkit for quickly performing crawler functions

Crawl the public data from Tefas.

Shared library for web crawler for map services

Basic web crawler

Yet Another Internet Carwler

A rock-solid cryprocurrency crawler

Boilerplate for developing crawler with Selenium.

51Degrees Device Detection parses HTTP headers to return detailed hardware, operating system, browser, and crawler information for the devices used to access your website or service. This package retrieves device detection results by consuming the 51Degrees cloud service.

Used for crawling arxiv papers automatically.

51Degrees Device Detection parses HTTP headers to return detailed hardware, operating system, browser, and crawler information for the devices used to access your website or service. This is an alternative to popular UAParser, DeviceAtlas, and WURFL packages.

Utils for stock-crawler project

The aiocrawler is a asynchronous/distributed web crawler/spider

Pull and standardize data on cloud compute resources.

a group of crawlers for private tracker website

Extracts information from different systems and convert them to dbt exposures

Crawler for Shopee from all country

Spidy is the simple, easy to use command line web crawler.