Python SDK for the Crawler Agent API
采集工具
Crawler for importing data from a filesystem directory into Solr
AI-powered geographic information crawler with multi-model support
Browserless crawler base: curl_cffi transport, TLS/AIA, identity, proxy, captcha, BaseCrawler/BaseParser.
Download lightnovels from 300+ sources and export to EPUB, MOBI, PDF, DOCX, and more — with a CLI, REST API, and web UI.
Python implementation Bloom filter
A python package that crawls a directory recursively and returns the file names and file stats as well as execute a custom function passed as an argument.
A distributed crawler framework based on Python
An image crawler, including multiple modules and GUI.
crawler_studio
Python SDK for the Scavio Search API - real-time data from 91 sources (Google, ChatGPT, Gemini, Amazon, Walmart, Costco, Best Buy, TCGplayer, Nintendo, H&M, Zara, Uniqlo, ASOS, Fashion Nova, Vinted, Mercari, Depop, Poshmark, thredUP, Macy's, IKEA, Wayfair, Lowe's, Nordstrom, Sephora, Ulta, ALDI, Walgreens, CVS, HEB, Kroger, Publix, Carrefour, Allegro, OLX, Mercado Libre, eBay, Zillow, Booking, Airbnb, Trustpilot, SEC EDGAR, Meta Ad Library and more) plus URL extraction, site map, crawl and batch extract
多站点产品数据爬虫 — 基于 DrissionPage,支持 Bass Pro / Walton's / Meat Your Maker 等站点
Crawl, extract and push climate metadata for indexing.
A light weight crawler which gives search results in HTML form or in Dictionary form, given urls and keywords.
A sample Crawler API
Open source tool to display/filter/export information about PCI or PCI Express devices, as well as their topology.
Crawling and feeding html content into a transmogrifier pipeline
Proxy rotation with PostgreSQL
LLM data collection and synthetic fine-tuning dataset pipeline
A stream crawler for ptt
This is a web application that extracts images URLs from web pages.
a collection of utils
A sample Crawler API
a hobby crawler
CrawlerX - The Ultimate Web Crawler
A client library for accessing crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page
Clark University, Package for YouTube crawler and cleaning data
Crawl through AWS accounts in an organization using master assumed role.
CrawlerDetect is a Python class for detecting bots/crawlers/spiders via the user agent.
Method integration package for web crawling.
tools which can easy the process of making crawler
The All in One Web Scraping Framework
Search for files or directories that match a set of rules
Python SDK for WebCrawler API
SELENIUM CRAWLER FOR SCRAPING BILLING DATA FROM AMOCRM PARTNER CABINET
Crawler para pegar telefones em sites. Totalmente amador e apenas de TESTE.
Multi-modal PyTorch dataloader using ds-crawler indices
Default weights for azcrawlerpy's reCAPTCHA v2 image-grid solver: TesseraNet's graphs, head and provenance card.
Python Test Crawler
This package allow you get data from Twitter, Swarm(check-ins) and Foursquare
A client to interact with freud-net API
Device Detection and User-Agent parser for Python that detects the device, browser, operating system, and crawlers from the User-Agent and User-Agent Client Hints, on-premise with a data file or in the cloud. An alternative to UAParser.js, DeviceAtlas, and WURFL.