Opensource Framework Crawler in Node.js
A library to test if a url(request) is crawled, usually used in a web crawler. Compatible with `request` and `node-crawler`
Low-level Rust-powered napi-rs bindings for Crawlee's filesystem storage. Not intended for direct use in crawlers — depend on @crawlee/fs-storage instead.
``` queueItem Object 当前爬取地址对象 queueItem.analysisUrlResult Array<Object> 分析html页面得出的url地址数组 queueItem.analysisResult Object 分析html页面得出的数据结果 queueIte
Production-grade, highly scalable web crawler and scraper with HTTP + browser fetching, Markdown extraction, CLI and HTTP API.
[![NPM version][npm-image]][npm-url] [![Build Status][travis-image]][travis-url] [![Dependency Status][depstat-image]][depstat-url] [![Downloads][download-badge]][npm-url]
Snapshot JavaScript-rendered pages with your own Chrome, check what crawlers actually see, and serve the snapshot to AI and search bots.
Server-side middleware that captures request data from AI crawlers and human traffic, sending events to the Searchable analytics pipeline.
Coherent, persistable anti-fingerprinting for Puppeteer. One seed → one consistent browser identity, with proxy/timezone/geo coherence and a fingerprint vault.
Fail CI when robots.txt blocks an AI answer crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot and more) from any page in your sitemap.
Script to monitor & download X (Twitter) Spaces 24/7
Beautiful-dom is a lightweight library that mirrors the capabilities of the HTML DOM API needed for parsing crawled HTML/XML pages. It models the methods and properties of HTML nodes that are relevant for extracting data from HTML nodes. It is written in
HTTP request module customized for crawlers.
爬虫调度程序
> 当前支持的模式
爬虫公共代码
Lightweight crawler written in TypeScript using ES6 generators.
Lightweight crawler written in TypeScript using ES6 generators.
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Node.js/TypeScript HTTP client with browser TLS fingerprint impersonation (JA3/JA4). Bypass Cloudflare and anti-bot detection. Rust-powered, fetch()-compatible.
Pure javascript cross-platform module to extract text from PDFs.
MCP server that gives AI agents the web: fetch, search, crawl and screenshot from one local Rust binary. Zero API keys, zero accounts, Chrome-true TLS.
爬虫管理界面
## 依赖
Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!
用于分析html文档中所需的数据
Sell crawl access to AI training crawlers by the day over x402, settled by CoinPay. One middleware: free bot development quotas, 402 offers, signed passes, Profullstack Power Keys, and robots.txt for search crawlers.
Pure javascript cross-platform module to extract page count from PDFs, based on pdf-parser.
duniter-crawler ===============
Fork of pdf-parse with fixes