Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously
Search for anything on web.
A library for efficiently walking a directory recursively
This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
Utilities to build Storybook crawling tools with Puppeteer
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
HTTP request module customized for crawlers.
Pure javascript cross-platform module to extract text from PDFs.
pure nodejs OPCUA SDK - module client-crawler
Stealth mode: Applies various techniques to make detection of headless puppeteer harder.
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
A library to recursively retrieve and serialize Notion pages with customization for machine learning applications.
[![npm](https://img.shields.io/npm/v/recrawl-sync.svg)](https://www.npmjs.com/package/recrawl-sync) [![ci](https://github.com/aleclarson/recrawl/actions/workflows/release.yml/badge.svg)](https://github.com/aleclarson/recrawl/actions/workflows/release.yml)
A web crawler that works with prember to discover URLs in your app
TypeScript definitions for crawler
Used to run a web crawler that checks for errors on specified pages.
A set of shared utilities that can be used by crawlers
Apify API client for JavaScript
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
Templates for the crawlee projects
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
TypeScript definitions for npm-license-crawler
An `URL` parser for crawling purpose.
The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.
TypeScript definitions for x-ray-crawler
SDK to interact with the web-crawler service
Automatically extracts structured information from webpages
A web crawler made for the SEO based on plugins. Please wait or contribute ... still in beta
Parse robot directives within HTML meta and/or HTTP headers.
TypeScript definitions for simplecrawler
A blazing fast recursive directory crawler with lazy sync and async iterator support.
Tool to crawl events, leagues and statistics from WBSC based websites.
Easily create XML sitemaps for your website.
A library to test if a url(request) is crawled, usually used in a web crawler. Compatible with `request` and `node-crawler`