A Python library for crawling THREDDS servers
Task Crawler and Reporter with Dashboard for Atlassian Confluence On-Prem Installations
Device Detection and User-Agent parser for Python that detects the device, browser, operating system, and crawlers from the User-Agent and User-Agent Client Hints using the 51Degrees cloud service. An alternative to UAParser.js, DeviceAtlas, and WURFL.
Device Detection and User-Agent parser for Python that detects the device, browser, operating system, and crawlers from the User-Agent and User-Agent Client Hints. This package holds the code shared by the cloud and on-premise engines for the 51Degrees Pipeline API. An alternative to UAParser.js, DeviceAtlas, and WURFL.
a test package
Intelligent Market Monitoring
Default weights for azcrawlerpy's text captcha solver: the SVTRv2 recogniser, its input-size selector and their provenance card.
Crawl telegra.ph searching for nudes!
CTFd-Crawler is a tool designed to efficiently manage and download CTF challenge files.
Help you to build web crawlers easily and quickly
crawler utils
This is the crawler libray
Spider templates for automatic crawlers.
Python package to query DeFi data from several The Graph subgraphs
A new crawler for LinkAhead
Device Detection and User-Agent parser for Python that detects the device, browser, operating system, and crawlers from the User-Agent and User-Agent Client Hints by reading a data file on your server. An alternative to UAParser.js, DeviceAtlas, and WURFL.
Composable HTTP, browser and proxy acquisition SDK for DataOS DataApps
Utils for stock-crawler project
DataLad extension package for crawling external web resources into an automated data distribution
Octoparse browser providers for DataApp Crawler
Generic packages with Crawlers for OSNs
Recursively finds tests in your package
A simple web-crawling framework, based on aiohttp.
A rather customizable image crawler structure, designed to download images with their information using multi-threading method. Besides, several wheels have been implemented to help better build a custom image crawler for yourself.
Python 3.x Web Crawler, Images, Urls, Emails, Phone numbers
Crawl your personal favorite images, photo albums, comics from website. Support pixiv, yande.re for now.
A small example package
Lint a robots.txt and audit it against 150 AI crawlers: which ones it really blocks, which rules do nothing, and which fetchers robots.txt cannot stop at all. RFC 9309, zero dependencies, no network.
this is an aparat crawler library
A web crawler CLI tool for preparing content for LLM ingestion and vector databases
update module import
a crawler script to extract and author metadata of spatial datasets
Automate downloads using predefined sites and the My-JDownloader-API
AI & LLM crawler readiness MCP server: audit robots.txt, validate llms.txt, and verify markdown mirrors for GEO/AEO optimization.
异步高并发dblp爬虫,慎用
Spidy is the simple, easy to use command line web crawler.
Boilerplate for developing crawler with Selenium.
LangChain integration for WebCrawlerAPI
Bilibili comment crawler with CLI, local analysis runs, and optional MCP stdio
Save a website or intranet as Markdown files, one per page (for search indexes and LLM tools)