Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.
Search for anything on web.
A library for efficiently walking a directory recursively
Detect Browser, Engine, OS, CPU, and Device type/model from User-Agent & Client Hints data. Supports browser & node.js environment
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
Transport-injected RSS, Atom, and JSON feed crawler.
HTML content decoding and extraction for crawlers.
Stealth mode: Applies various techniques to make detection of headless puppeteer harder.
Utilities to build Storybook crawling tools with Puppeteer
A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.
HTTP request module customized for crawlers.
Find broken links, missing images, etc in your HTML. Scurry around your site and find all those broken links.
In-memory route crawling and static path discovery for Remix.
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
Node.js/TypeScript HTTP client with browser TLS fingerprint impersonation (JA3/JA4). Bypass Cloudflare and anti-bot detection. Rust-powered, fetch()-compatible.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
[](https://www.npmjs.com/package/recrawl-sync) [](https://github.com/aleclarson/recrawl/actions/workflows/release.yml)
Detect Browser, Engine, OS, CPU, and Device type/model from User-Agent & Client Hints data. Supports browser & node.js environment
pure nodejs OPCUA SDK - module client-crawler (unmaintained, still MIT)
Utilities to build Storybook crawling tools with Puppeteer
Promptbook: Create persistent AI agents that turn your company's scattered knowledge into action
Detect Browser, Engine, OS, CPU, and Device type/model from User-Agent & Client Hints data. Supports browser & node.js environment
JavaScript module detecting bots/crawlers/spiders via user-agent
A polite web crawler for research and AI agents. It fetches a source once, keeps it, and reads it many times: robots.txt obeyed, one request per site, full-text search. The MCP server is the package polite-crawler-mcp.
Node.js/TypeScript HTTP client with browser TLS fingerprint impersonation (JA3/JA4). Bypass Cloudflare and anti-bot detection. Rust-powered, fetch()-compatible.
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more) can access your site: robots.txt per bot, live HTTP status as each bot, robots.txt generator, CLI for CI and an MCP server.
The MCP server of polite-crawler: nine tools that fetch web pages politely, keep them, search them and verify quotes, for Claude and other MCP clients.
A web crawler made for the SEO based on plugins. Please wait or contribute ... still in beta