Crawler

Search for anything on web.

A library for efficiently walking a directory recursively

[](https://www.npmjs.com/package/recrawl-sync) [](https://github.com/aleclarson/recrawl/actions/workflows/release.yml)

Stealth mode: Applies various techniques to make detection of headless puppeteer harder.

🤖 detect bots/crawlers/spiders via the user agent.

Pure javascript cross-platform module to extract text from PDFs.

Yet another node torrent scraper based on x-ray. (Support iptorrents, torrentleech, torrent9, Yyggtorrent, ThePiratebay, torrentz2, 1337x, KickassTorrent, Rarbg, TorrentProject, Yts, Limetorrents, Eztv)

The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s

Node.js agent for Sqreen, please see https://www.sqreen.io/

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Analyzes license information for multiple node.js modules (package.json files) as part of your software project.

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

HTTP request module customized for crawlers.

Parse robot directives within HTML meta and/or HTTP headers.

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously

Apify API client for JavaScript

A library to test if a url(request) is crawled, usually used in a web crawler. Compatible with `request` and `node-crawler`

pure nodejs OPCUA SDK - module -client-crawler

This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

[](https://www.npmjs.com/package/recrawl) [](https://github.com/aleclarson/recrawl/actions/workflows/release.yml) [![codeco

A set of shared utilities that can be used by crawlers

Easily create XML sitemaps for your website.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Automatically extracts structured information from webpages

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

Html Metadata scraper and parser for Node.js

Templates for the crawlee projects

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

JavaScript module detecting bots/crawlers/spiders via user-agent

x-ray's crawler

Extract data from a pdf with pure javascript

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Pure javascript cross-platform module to extract page count from PDFs, based on pdf-parser.

A blazing fast recursive directory crawler with lazy sync and async iterator support.

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Verify that a request is from Google using Google's recommended DNS verification steps

Recursive directory reader with a delightful API

A web crawler that works with prember to discover URLs in your app

Create xml sitemaps from the command line.

Headless Chrome abstraction to simplify the interaction with the browser. It may be used for crawling sites, test automation, etc

http client module with cheerio & iconv(-lite) & promise

Pure javascript cross-platform module to extract text from PDFs.