Crawler

A configuration - based crawler framework

Functions used to transform, create, extract info on URIs

A library for checking basic SEO signals of a website

An easiest crawling and scraping module for NestJS

Library for crawling across the Banano ledger to trace NFTs.

Crawl web as easy as possible

JavaScript SDK for Firecrawl API

基于Node.js的网络爬虫(轻量版)，该版本不支持动态页面爬取。

Node.js Hydra web crawler

A JavaScript library that allows for the quick transformation of DOM documents into useful formats.

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

Distributed web crawler powered by Headless Chrome

A port of n0madic/twitter-scraper to Node.js.

Take a snapshot of any website.

http client module with cheerio & iconv(-lite) & promise

 ============

A list of common crawler agents used on Internet..

Automatically extracts structured information from webpages

A robust GitHub API crawler that walks a queue of GitHub entities retrieving and storing their contents.

Express middleware that informs if request originates from a Google Bot or a Google Crawler

Bright CLI is a CLI tool that can initialize, stop, polling and maintain scans in Bright solutions.

Node Web Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!

`crsp` is a command-line interface (CLI) for creating, running, and deploying web crawlers to [crawlspace.dev](https://crawlspace.dev).

Declarative and Observable Distributed Crawler For Web, RDB, OS, also can act as a Monitor or ETL for your system

crawler company

Helper functions to crawl web pages from search results.

This is personal project for web crawling/scraping topics. It includes few ways to crawl the data mainly using [Node.js](https://nodejs.org/en/) such as:

## services

```javascript

This is personal project for web crawling/scraping topics. It includes few ways to crawl the data mainly using [Node.js](https://nodejs.org/en/) such as:

download images from calmara.com

Prerendering for Single Page Applications to improve SEO

Phâm tích html

面试经验爬虫

Expired domains finder for crawler.ninja

爬虫管理界面

用于发现html文档中的地址链接

the worker side of nova-crawler

大厂面经爬虫

The best module ever.

Redis store for Crawler.ninja

crawler util

crawler service

爬虫公共代码

Crawler UI Dashboard

[![NPM version][npm-image]][npm-url] [![Build Status][travis-image]][travis-url] [![Dependency Status][depstat-image]][depstat-url] [![Downloads][download-badge]][npm-url]