Crawler v2 : Advanced and Typescript version of node-crawler
Features:
Requires Node.js 16 or above
$ npm install crawler
Warning: Given the dependencies involved (Especially migrating from request to got) , Crawler v2 has been designed as a native ESM and no longer offers a CommonJS export. We would also like to recommend that you convert to ESM. Note that making this transition is generally not too difficult.
Support both Promise and callback
import Crawler from "../node-crawler/dist/index.js";
const crawler = new Crawler();
// When using directly "send", the preRequest won't be called and the "Event:request" won't be triggered
const response = await crawler.send("https://github.com/");
console.log(response.options);
// console.log(response.body);
crawler.send({
url: "https://github.com/",
// When calling `send`, `callback` must be defined explicitly, with two arguments `error` and `response`
callback: (error, response) => {
if (error) {
console.error(error);
} else {
console.log("Hello World!");
}
},
});
import Crawler from "crawler";
const c = new Crawler({
maxConnections: 10,
// This will be called for each crawled page
callback: (error, res, done) => {
if (error) {
console.log(error);
} else {
const $ = res.$;
// $ is Cheerio by default
//a lean implementation of core jQuery designed specifically for the server
console.log($("title").text());
}
done();
},
});
// Add just one URL to queue, with default callback
c.add("http://www.amazon.com");
// Add a list of URLs
c.add(["http://www.google.com/", "http://www.yahoo.com"]);
// Add URLs with custom callbacks & parameters
c.add([
{
url: "http://parishackers.org/",
jQuery: false,
// The global callback won't be called
callback: (error, res, done) => {
if (error) {
console.log(error);
} else {
console.log("Grabbed", res.body.length, "bytes");
}
done();
},
},
]);
// Add some HTML code directly without grabbing (mostly for tests)
c.add([
{
html: "<title>This is a test</title>",
},
]);
please refer to options for detail.
Use rateLimit to slow down when you are visiting web sites.
import Crawler from "crawler";
const c = new Crawler({
rateLimit: 1000, // `maxConnections` will be forced to 1
callback: (err, res, done) => {
console.log(res.$("title").text());
done();
},
});
c.add(tasks); //between two tasks, minimum time gap is 1000 (ms)
Sometimes you have to access variables from previous request/response session, what should you do is passing parameters in options.userParams :
c.add({
url: "http://www.google.com",
userParams: {
parameter1: "value1",
parameter2: "value2",
parameter3: "value3",
},
});
then access them in callback via res.options
console.log(res.options.userParams);
If you are downloading files like image, pdf, word etc, you have to save the raw response body which means Crawler shouldn't convert it to string. To make it happen, you need to set encoding to null
import Crawler from "crawler";
import fs from "fs";
const c = new Crawler({
encoding: null,
jQuery: false, // set false to suppress warning message.
callback: (err, res, done) => {
if (err) {
console.error(err.stack);
} else {
fs.createWriteStream(res.options.userParams.filename).write(res.body);
}
done();
},
});
c.add({
url: "https://raw.githubusercontent.com/bda-research/node-crawler/master/crawler_primary.png",
userParams: {
filename: "crawler.png",
},
});
If you want to do something either synchronously or asynchronously before each request, you can try the code below. Note that direct requests won't trigger preRequest.
import Crawler from "crawler";
const c = new Crawler({
preRequest: (options, done) => {
// 'options' here is not the 'options' you pass to 'c.queue', instead, it's the options that is going to be passed to 'request' module
console.log(options);
// when done is called, the request will start
done();
},
callback: (err, res, done) => {
if (err) {
console.log(err);
} else {
console.log(res.statusCode);
}
},
});
c.add({
url: "http://www.google.com",
// this will override the 'preRequest' defined in crawler
preRequest: (options, done) => {
setTimeout(() => {
console.log(options);
done();
}, 1000);
},
});
Now we offer hassle-free support for using HTTP/2: just set http2 to true, and Crawler will operate as smoothly as with HTTP (including proxies).
Note: As most developers using this library with proxies also work with Charles, it is expected to set rejectAuthority to false in order to prevent the so-called 'self-signed certificate' errors."
crawler.send({
url: "https://nghttp2.org/httpbin/status/200",
method: "GET",
http2: true,
callback: (error, response) => {
if (error) {
console.error(error);
}
console.log(`inside callback`);
console.log(response.body);
},
});
Control the rate limit. All tasks submit to a rateLimiter will abide the rateLimit and maxConnections restrictions of the limiter. rateLimit is the minimum time gap between two tasks. maxConnections is the maximum number of tasks that can be running at the same time. rateLimiters are independent of each other. One common use case is setting different rateLimiters for different proxies. One thing is worth noticing, when rateLimit is set to a non-zero value, maxConnections will be forced to 1.
import Crawler from "crawler";
const c = new Crawler({
rateLimit: 2000,
maxConnections: 1,
callback: (error, res, done) => {
if (error) {
console.log(error);
} else {
const $ = res.$;
console.log($("title").text());
}
done();
},
});
// if you want to crawl some website with 2000ms gap between requests
c.add("http://www.somewebsite.com/page/1");
c.add("http://www.somewebsite.com/page/2");
c.add("http://www.somewebsite.com/page/3");
// if you want to crawl some website using proxy with 2000ms gap between requests for each proxy
c.add({
url: "http://www.somewebsite.com/page/1",
rateLimiterId: 1,
proxy: "proxy_1",
});
c.add({
url: "http://www.somewebsite.com/page/2",
rateLimiterId: 2,
proxy: "proxy_2",
});
c.add({
url: "http://www.somewebsite.com/page/3",
rateLimiterId: 3,
proxy: "proxy_3",
});
c.add({
url: "http://www.somewebsite.com/page/4",
rateLimiterId: 4,
proxy: "proxy_1",
});
Normally, all ratelimiters instances in the limiter cluster of crawler are instantiated with options specified in crawler constructor. You can change property of any rateLimiter by calling the code below. Currently, we only support changing property 'rateLimit' of it. Note that the default rateLimiter can be accessed by crawler.setLimiter(0, "rateLimit", 1000);. We strongly recommend that you leave limiters unchanged after their instantiation unless you know clearly what you are doing.
const crawler = new Crawler();
crawler.setLimiter(0, "rateLimit", 1000);
optionsEmitted when a task is being added to scheduler.
crawler.on("schedule", options => {
options.proxy = "http://proxy:port";
});
optionsrateLimiterId : numberEmitted when limiter has been changed.
optionsEmitted when crawler is ready to send a request.
If you are going to modify options at last stage before requesting, just listen on it.
crawler.on("request", options => {
options.searchParams.timestamp = new Date().getTime();
});
Emitted when queue is empty.
crawler.on("drain", () => {
// For example, release a connection to database.
db.end(); // close connection to MySQL
});
url | optionsAdd a task to queue and wait for it to be executed.
NumberSize of queue, read-only
You can pass these options to the Crawler() constructor if you want them to be global or as items in the crawler.add() calls if you want them to be specific to that item (overwriting global options)
maxConnectionsnumberpriorityLevelsnumberrateLimitType: number
Default : 0
1000 means 1000 milliseconds delay between after the first request.
Note: This options is list as global only options because it will be set as the "default rateLimit value". This value is bound to a specific rate limiter and can only be modified through the crawler.setLimiter method. Please avoid passing redundant rateLimit property in local requests; instead, use options.rateLimiterId to specify a particular limiter.
Example:
crawler.on("schedule", options => {
options.rateLimiterId = Math.floor(Math.random() * 15);
});
skipDuplicatesbooleanhomogeneousbooleanuserAgentsstring | string[]url | method | headers | body | searchParams...forceUTF8booleanjQuerybooleanencodingstringrateLimiterIdnumberretriesnumberretryIntervalnumbertimeoutnumberprioritynumberskipEventRequestbooleanhtmlbooleanproxiesstring[]const ProxyManager = {
index: 0,
proxies: JSON.parse(fs.readFileSync("../proxies.json")),
setProxy: function (options) {
let proxy = this.proxies[this.index];
this.index = ++this.index % this.proxies.length;
options.proxy = proxy;
options.rateLimiterId = Math.floor(Math.random() * 15);
},
};
crawler.on("schedule", options => {
// options.proxy = "http://127.0.0.1:8000";
ProxyManager.setProxy(options);
});
proxystringhttp2booleanrefererstringuserParamsunknownres.options.preRequest(options, done) => unknowncrawler.add method.CallbackType: (error, response, done) => unknown
Function that will be called after a request was completed
error: Error catched by the crawlerresponse : http.IncomingMessage A response of standard IncomingMessage includes $ and options
res.options: Options of this taskres.$: jQuery Selector A selector for html or xml document.res.statusCode: Number HTTP status code. E.G.200res.body: Buffer | String HTTP response content which could be a html page, plain text or xml document e.g.res.headers: Object HTTP response headersres.request: Request An instance of Mikeal's Request instead of http.ClientRequest
done : The function must be called when you've done your work in callback. This is the only way to tell the crawler that the task is finished.Crawler by default use Cheerio. We are temporarily no longer supporting jsdom for certain reasons.
Crawler uses nock to mock http request, thus testing no longer relying on http server.
$ pnpm test
FAQs
Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.
The npm package crawler receives a total of 2,959 weekly downloads. As such, crawler popularity was classified as popular.
We found that crawler demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.