Crawler

Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.

A library for efficiently walking a directory recursively

Search for anything on web.

The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s

This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.

Utilities to build Storybook crawling tools with Puppeteer

🤖/👨‍🦰 Recognise bots/crawlers/spiders using the user agent string.

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

Pure javascript cross-platform module to extract text from PDFs.

HTTP request module customized for crawlers.

A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.

pure nodejs OPCUA SDK - module client-crawler (deprecated - use @sterfive/crawler module instead)

Stealth mode: Applies various techniques to make detection of headless puppeteer harder.

Analyzes license information for multiple node.js modules (package.json files) as part of your software project.

[](https://www.npmjs.com/package/recrawl-sync) [](https://github.com/aleclarson/recrawl/actions/workflows/release.yml)

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

TypeScript definitions for simplecrawler

JavaScript SDK for Firecrawl API

An `URL` parser for crawling purpose.

Async and sync crawler for json object

x-ray's crawler

Apify API client for JavaScript

A web crawler that works with prember to discover URLs in your app

Used to run a web crawler that checks for errors on specified pages.

A set of shared utilities that can be used by crawlers

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Templates for the crawlee projects

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

TypeScript definitions for crawler

blocklet crawler lib

JavaScript module detecting bots/crawlers/spiders via user-agent

Parse robot directives within HTML meta and/or HTTP headers.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

TypeScript definitions for x-ray-crawler

Automatically extracts structured information from webpages

Analyzes license information for multiple node.js modules (package.json files) as part of your software project.

爬虫下载插件

This plugin links your Netlify site with Algolia's Crawler. It will trigger a crawl on each successful build.

A web crawler made for the SEO based on plugins. Please wait or contribute ... still in beta

MCP server for Firecrawl web scraping integration. Supports both cloud and self-hosted instances. Features include web scraping, batch processing, structured data extraction, and LLM-powered content analysis.

A web scraper for NodeJs