x-crawl - npm Package Compare versions

Comparing version 8.3.0 to 8.3.1

package.json

 {
   "name": "x-crawl",
-  "version": "8.3.0",
+  "version": "8.3.1",
   "author": "coderHXL",
@@ -43,4 +43,4 @@ "description": "x-crawl is a flexible Node.js multifunctional crawler library.",
     "https-proxy-agent": "^7.0.1",
-    "puppeteer": "21.1.0"
+    "puppeteer": "21.6.1"
   }
 }

README.md

@@ -134,2 +134,3 @@ # x-crawl · [![npm](https://img.shields.io/npm/v/x-crawl.svg)](https://www.npmjs.com/package/x-crawl) [![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/coder-hxl/x-crawl/blob/main/LICENSE)
   - [The relationship between crawlPage API and puppeteer](#the-relationship-between-crawlpage-api-and-puppeteer)
+  - [Using crawlPage API causes the program to crash](#using-crawlpage-api-causes-the-program-to-crash)
 - [More](#more)
@@ -338,4 +339,2 @@ - [Community](#community)
 
-**Note:** If you need to crawl many pages at one time, you need to use this life cycle function to process the results of each target and close the page instance after each page is crawled down. If you do not close the page instance, then The program will crash due to too many opened pages.
-
 #### Open Browser
@@ -2077,2 +2076,29 @@
 
+### Using crawlPage API causes the program to crash
+
+If you need to crawl many pages in one crawlPage, it is recommended that after crawling each page, use [onCrawlItemComplete life cycle function] (#onCrawlItemComplete) to process the results of each target and close the page instance. If no shutdown operation is performed, then The program may crash due to too many pages being opened (related to the performance of the device itself).
+
+```js
+import xCrawl from 'x-crawl'
+
+const myXCrawl = xCrawl()
+
+// Use the advanced configuration mode
+myXCrawl.crawlPage({
+  targets: [
+    'https://www.example.com/page-1',
+    'https://www.example.com/page-2',
+    'https://www.example.com/page-3',
+    'https://www.example.com/page-4',
+    'https://www.example.com/page-5',
+    'https://www.example.com/page-6'
+  ],
+  onCrawlItemComplete(crawlPageSingleResult) {
+    const { page } = crawlPageSingleResult.data
+
+    page.close()
+  }
+})
+```
+
 ## More
@@ -2079,0 +2105,0 @@

No alert changes

Improved metrics

Total package byte prevSize

145490

increased by0.51%

Number of lines in readme file

2122

increased by1.24%

Dependency changes

• + Added@puppeteer/browsers@1.9.0(transitive)

• + Addedchromium-bidi@0.5.1(transitive)

• + Addedcosmiconfig@8.3.6(transitive)

• + Addeddevtools-protocol@0.0.1203626(transitive)

• + Addedproxy-agent@6.3.1(transitive)

• + Addedpuppeteer@21.6.1(transitive)

• + Addedpuppeteer-core@21.6.1(transitive)

• + Addedurlpattern-polyfill@9.0.0(transitive)

• + Addedws@8.15.1(transitive)

• + Addedyargs@17.7.2(transitive)

• - Removed@puppeteer/browsers@1.7.0(transitive)

• - Removedchromium-bidi@0.4.20(transitive)

• - Removedcosmiconfig@8.2.0(transitive)

• - Removeddevtools-protocol@0.0.1159816(transitive)

• - Removedproxy-agent@6.3.0(transitive)

• - Removedpuppeteer@21.1.0(transitive)

• - Removedpuppeteer-core@21.1.0(transitive)

• - Removedws@8.13.0(transitive)

• - Removedyargs@17.7.1(transitive)

• Updatedpuppeteer@21.6.1