New Case Study:See how Anthropic automated 95% of dependency reviews with Socket.Learn More →
crawler - npm Package Compare versions
Comparing version 2.0.0 to 2.0.1
| { | ||
| "name": "crawler", | ||
| "version": "2.0.0", | ||
| "description": "Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and seamless HTTP/2 support.", | ||
| "version": "2.0.1", | ||
| "description": "Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.", | ||
| "repository": { | ||
@@ -12,3 +12,4 @@ "type": "git", | ||
| "build": "tsc", | ||
| "test": "mocha" | ||
| "test": "ava", | ||
| "cover": "c8 ava" | ||
| }, | ||
@@ -28,9 +29,9 @@ "engines": { | ||
| "http", | ||
| "https", | ||
| "http2", | ||
| "got", | ||
| "https", | ||
| "http2", | ||
| "got", | ||
| "request", | ||
| "url", | ||
| "network", | ||
| "gzip" | ||
| "url", | ||
| "network", | ||
| "gzip" | ||
| ], | ||
@@ -40,3 +41,3 @@ "license": "MIT", | ||
| "cheerio": "1.0.0-rc.12", | ||
| "got": "^14.4.0", | ||
| "got": "^14.4.1", | ||
| "hpagent": "^1.2.0", | ||
@@ -46,27 +47,39 @@ "http2-wrapper": "^2.2.1", | ||
| "seenreq": "^3.0.0", | ||
| "tslog": "^4.9.2" | ||
| "sinon": "^18.0.0", | ||
| "tslog": "^4.9.3" | ||
| }, | ||
| "devDependencies": { | ||
| "@eslint/js": "^9.4.0", | ||
| "@eslint/js": "^9.5.0", | ||
| "@types/got": "^9.6.12", | ||
| "@types/node": "^20.14.2", | ||
| "chai": "^5.1.1", | ||
| "@types/node": "^20.14.7", | ||
| "ava": "^6.1.3", | ||
| "c8": "^10.1.2", | ||
| "eslint": "~9.4.0", | ||
| "globals": "^15.3.0", | ||
| "mocha": "^10.4.0", | ||
| "mocha-testdata": "^1.2.0", | ||
| "globals": "^15.6.0", | ||
| "nock": "^13.5.4", | ||
| "sinon": "^17.0.1", | ||
| "tough-cookie": "^4.1.4", | ||
| "tsx": "^4.11.2", | ||
| "typescript": "^5.4.5", | ||
| "tsx": "^4.15.7", | ||
| "typescript": "^5.5.2", | ||
| "typescript-eslint": "8.0.0-alpha.27" | ||
| }, | ||
| "mocha": { | ||
| "ava": { | ||
| "files": [ | ||
| "test/**/*.js" | ||
| "test/*.js", | ||
| "!test/*test.js" | ||
| ], | ||
| "timeout": 15000, | ||
| "exit": true | ||
| "timeout": "20s", | ||
| "extensions": { | ||
| "js": true | ||
| }, | ||
| "failFast": true, | ||
| "verbose": true | ||
| }, | ||
| "c8": { | ||
| "reporter": [ | ||
| "lcov", | ||
| "html", | ||
| "text" | ||
| ], | ||
| "clean": true | ||
| } | ||
| } |
@@ -7,6 +7,7 @@ <p align="center"> | ||
| # | ||
| ------ | ||
| [](https://www.npmjs.com/package/crawler/v/2.0.0-beta.5) | ||
| [](https://www.npmjs.com/package/crawler/v/2.0.0) | ||
| [](https://circleci.com/gh/bda-research/node-crawler/tree/master) | ||
| [![NPM download][download-image]][download-url] | ||
@@ -36,4 +37,6 @@ [![Package Quality][quality-image]][quality-url] | ||
| Requires Node.js 18 or above | ||
| **IMPORTANT:** If you are using a Linux OS, we currently recommend sticking with Node.js version 18 for the time being, rather than opting for higher versions (even if some dependencies suggest 20 or later). Our unit tests have encountered stability issues on Linux with higher versions of Node.js, which may be caused by more profound underlying reasons. However, at present, we do not have the resources to address these issues. | ||
| Requires Node.js 18 or above. | ||
| **IMPORTANT:** If you are using a Linux OS, we currently recommend sticking with **Node.js version 18** for the time being, rather than opting for higher versions (even if some dependencies suggest 20 or later). Our unit tests have encountered stability issues on Linux with higher versions of Node.js, which may be caused by more profound underlying reasons. However, at present, we do not have the resources to address these issues. | ||
| ```sh | ||
@@ -47,29 +50,2 @@ $ npm install crawler | ||
| ### Direct request | ||
| Support both Promise and callback | ||
| ```js | ||
| import Crawler from "crawler"; | ||
| const crawler = new Crawler(); | ||
| // When using directly "send", the preRequest won't be called and the "Event:request" won't be triggered | ||
| const response = await crawler.send("https://github.com/"); | ||
| console.log(response.options); | ||
| // console.log(response.body); | ||
| crawler.send({ | ||
| url: "https://github.com/", | ||
| // When calling `send`, `callback` must be defined explicitly, with two arguments `error` and `response` | ||
| callback: (error, response) => { | ||
| if (error) { | ||
| console.error(error); | ||
| } else { | ||
| console.log("Hello World!"); | ||
| } | ||
| }, | ||
| }); | ||
| ``` | ||
| ### Execute asynchronously via custom options | ||
@@ -233,4 +209,31 @@ | ||
| ### Direct request | ||
| Support both Promise and callback | ||
| ```js | ||
| import Crawler from "crawler"; | ||
| const crawler = new Crawler(); | ||
| // When using directly "send", the preRequest won't be called and the "Event:request" won't be triggered | ||
| const response = await crawler.send("https://github.com/"); | ||
| console.log(response.options); | ||
| // console.log(response.body); | ||
| crawler.send({ | ||
| url: "https://github.com/", | ||
| // When calling `send`, `callback` must be defined explicitly, with two arguments `error` and `response` | ||
| callback: (error, response) => { | ||
| if (error) { | ||
| console.error(error); | ||
| } else { | ||
| console.log("Hello World!"); | ||
| } | ||
| }, | ||
| }); | ||
| ``` | ||
| ### | ||
| # Table | ||
@@ -237,0 +240,0 @@ |
No alert changes
Improved metrics
- Dev dependency count
- decreased by-14.29%
12
- Number of lines in readme file
- increased by0.45%
674
Worsened metrics
- Total package byte prevSize
- decreased by-17.84%
106253
- Dependency count
- increased by14.29%
8
- Number of package files
- decreased by-3.51%
55
- Lines of code
- decreased by-2.86%
1326
Dependency changes
+ Addedsinon@^18.0.0
+ Added@sinonjs/commons@3.0.1(transitive)
+ Added@sinonjs/fake-timers@11.2.213.0.5(transitive)
+ Added@sinonjs/samsam@8.0.2(transitive)
+ Added@sinonjs/text-encoding@0.7.3(transitive)
+ Addeddiff@5.2.0(transitive)
+ Addedhas-flag@4.0.0(transitive)
+ Addedjust-extend@6.2.0(transitive)
+ Addedlodash.get@4.4.2(transitive)
+ Addednise@6.1.1(transitive)
+ Addedpath-to-regexp@8.2.0(transitive)
+ Addedsinon@18.0.1(transitive)
+ Addedsupports-color@7.2.0(transitive)
+ Addedtype-detect@4.0.84.1.0(transitive)
Updatedgot@^14.4.1
Updatedtslog@^4.9.3