Web Scraping Framework based on py3 asyncio
CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.
Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters.
Crawlera middleware for Scrapy
Python Test Crawler
crawler commons
Class that provides decorators and functions for easy handling of crawlera sessions in a scrapy spider.
An app to download novels from online sources and generate e-books.
A client implementation of Firefox DevTools over remote debug protocol.
Command-line program to download image galleries and collections from several image hosting sites
A shared library for web scraping utilities.
Core programs for crawling
Framework for crawling
Browser fingerprint datapoints collected by Apify
Open source tool to display/filter/export information about PCI or PCI Express devices, as well as their topology.
Python SDK for WebCrawler API
Python implementation Bloom filter
A web Crawler for DTC(dans ton chat), VDM(vie de merde) and SCMB(se coucher moins bete)
采集工具
A distributed crawler framework based on Python
Automate downloads using predefined sites and the My-JDownloader-API
SELENIUM CRAWLER FOR SCRAPING BILLING DATA FROM AMOCRM PARTNER CABINET
this is an aparat crawler library
crawler utils
Scrapy utils for Modis crawlers projects.
CrawlerDetect is a Python class for detecting bots/crawlers/spiders via the user agent.
Python project to automate extracting job information directly from ATS
Python package to detect bots/crawlers/spiders via user-agent
LangChain integration for WebCrawlerAPI
A modular, async Python framework for structured online data collection used by Ollama Agent Roll Cage (OARC)
A sample Crawler API
Autonomous Scraping Agent: Scrape URLs with prompts and schema
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page
This is the crawler libray
A python library for extracting data from html table
Crawl the public data from Tefas.
Video Crawler
A sample Crawler API
This is a web application that extracts images URLs from web pages.
Intelligent Market Monitoring
51Degrees Device Detection parses HTTP headers to return detailed hardware, operating system, browser, and crawler information for the devices used to access your website or service. This package retrieves device detection results by consuming the 51Degrees cloud service.
Configurable crawler for web-scraping