A Node.js SDK for integrating crawlergo, xray, and req probe tools
## 下载htlm文档中间件
The best module ever.
the worker side of nova-crawler
download images from calmara.com
用于发现html文档中的地址链接
Distributed web crawler powered by Headless Chrome
Node.js/TypeScript HTTP client with browser TLS fingerprint impersonation (JA3/JA4). Bypass Cloudflare and anti-bot detection. Rust-powered, fetch()-compatible.
A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.
ParallaxAPIs SDK
Turn web pages into clean Markdown for LLMs: a polite, Firecrawl-compatible scraping engine written in Rust.
Compile robots.txt rules for Google-compatible or strict RFC 9309 matching
Keep Bots Away From Your Express App
Low-level Rust-powered napi-rs bindings for Crawlee's filesystem storage. Not intended for direct use in crawlers — depend on @crawlee/fs-storage instead.
Turn web pages into clean Markdown for LLMs: a polite, Firecrawl-compatible scraping engine written in Rust.
Node.js/TypeScript HTTP client with browser TLS fingerprint impersonation (JA3/JA4). Bypass Cloudflare and anti-bot detection. Rust-powered, fetch()-compatible.
Pure javascript cross-platform module to extract text from PDFs.
Website QA tool built for coding agents. 350+ rules across SEO, performance, security, accessibility & agent experience, from the CLI, cloud, or MCP.
Policy-first crawler control for Astro — generates robots.txt and llms.txt with presets, per-bot rules, AI crawler registry, and build-time audits.
Turn web pages into clean Markdown for LLMs: a polite, Firecrawl-compatible scraping engine written in Rust.
Solid's integration for prerender-crawler: prerendered() turns server functions into build-time data captured as static artifacts, and the serverFunctions() integration captures them during the crawl and guards static builds against calls nothing prerende
Turn web pages into clean Markdown for LLMs: a polite, Firecrawl-compatible scraping engine written in Rust.
Bot detection and AI agent verification for Node.js and edge runtimes. Local honeypot tripwires, rate limiting and AI crawler policies, with optional WebDecoy dashboard reporting.
The official TypeScript library for the API Crawler Dev SDKs API
* Install the package globally.
Web crawler library
爬虫任务调度器的JsSDK
Crawler Second-system effect,the second development
Crawl elements selector which match specific DOM event
Phâm tích html
HTTP request module customized for crawlers.
Crawler4nodejs is an open source web crawler for Node.js which provides a simple interface for crawling the Web.
## services
crawler lib for nodejs