x-crawl

x-crawl

English | 简体中文

x-crawl is a flexible nodejs crawler library. It can crawl pages in batches, network requests in batches, download file resources in batches, polling and crawling, etc. Flexible and simple to use, friendly to JS/TS developers.

If you like x-crawl, you can give x-crawl repository a star to support it, not only for its recognition, but also for Approved by the developer.

Features

• 🔥 Async/Sync - Just change the mode property to toggle async/sync crawling mode.
• ⚙️ Multiple functions - Batch crawling of pages, batch network requests, batch download of file resources, polling crawling, etc.
• 🖋️ Flexible writing method - A function adapts to multiple crawling configurations and obtains crawling results. The writing method is very flexible.
• ⏱️ Interval crawling - no interval/fixed interval/random interval, can effectively use/avoid high concurrent crawling.
• 🔄 Retry on failure - It can be set for all crawling requests, for a single crawling request, and for a single request to set a failed retry.
• 🚀 Priority Queue - Use priority crawling based on the priority of individual requests.
• ☁️ Crawl SPA - Batch crawl SPA (Single Page Application) to generate pre-rendered content (ie "SSR" (Server Side Rendering)).
• ⚒️ Controlling Pages - Headless browsers can submit forms, keystrokes, event actions, generate screenshots of pages, etc.
• 🧾 Capture Record - Capture and record the crawled results, and highlight them on the console.
• 🦾TypeScript - Own types, implement complete types through generics.

Relationship with puppeteer

The crawlPage API internally uses the puppeteer library to help us crawl pages and expose Brower instances and Page instances.

Table of Contents

• Install
• Example
• Core concepts
  • Create application
    • An example of a crawler application
    • Choose crawling mode
    • Multiple crawler application instances
  • Crawl page
    • browser instance
    • page instance
  • Crawl interface
  • Crawl files
  • Start polling
  • Config priority
  • Interval time
  • Fail retry
  • Priority queue
  • [About the results](#About the results)
• API
  • xCrawl
    • Type
    • Example
  • crawlPage
    • Type
    • Example
    • Config
  • crawlData
    • Type
    • Example
    • Config
  • crawlFile
    • Type
    • Example
    • Config
  • crawlPolling
    • Type
    • Example
• Types
  • API Config
    • IntervalTime
    • Method
    • PageRequestConfigCookies
    • PageRequestConfig
    • DataRequestConfig
    • FileRequestConfig
    • CrawlPageConfigObject
    • CrawlDataConfigObject
    • CrawlFileConfigObject
    • XCrawlBaseConfig
    • CrawlPageConfig
    • CrawlDataConfig
    • CrawlFileConfig
    • StartPollingConfig
    • API Result
      • XCrawlInstance
      • CrawlCommonRes
      • CrawlPageSingleRes
      • CrawlDataSingleRes
      • CrawlFileSingleRes
      • CrawlPageRes
      • CrawlDataRes
      • CrawlFileRes
    • API Other
      • AnyObject
• More

Install

Take NPM as an example:

npm install x-crawl

Example

Timing capture: Take the automatic capture of the cover image of Airbnb Plus listings every day as an example:

// 1.Import module ES/CJS
import xCrawl from 'x-crawl'

// 2.Create a crawler instance
const myXCrawl = xCrawl({ intervalTime: { max: 3000, min: 2000 } })

// 3.Set the crawling task
/*
  Call the startPolling API to start the polling function,
  and the callback function will be called every other day
*/
myXCrawl.startPolling({ d: 1 }, async (count, stopPolling) => {
  // Call crawlPage API to crawl Page
  const res = await myXCrawl.crawlPage('https://zh.airbnb.com/s/*/plus_homes')
  const { page } = res.data

  // set request configuration
  const plusBoxHandle = await page.$('.a1stauiv')
  const requestConfigs = await plusBoxHandle!.$$eval(
    'picture img',
    (imgEls) => {
      return imgEls.map((item) => item.src)
    }
  )

  // Call the crawlFile API to crawl pictures
  myXCrawl.crawlFile({ requestConfigs, fileConfig: { storeDir: './upload' } })

  // Close page
  page.close()
})

running result:

Note: Do not crawl at will, you can check the robots.txt protocol before crawling. This is just to demonstrate how to use x-crawl.

Core concepts

Create application

An example of a crawler application

Create a new application instance via xCrawl():

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  // options
})

Related options can refer to XCrawlBaseConfig .

Choose crawling mode

A crawler application instance has two crawling modes: asynchronous/synchronous, and each crawler instance can only choose one of them.

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  mode: 'async'
})

The mode option defaults to async .

• async: asynchronous request, in batch requests, the next request is made without waiting for the current request to complete
• sync: synchronous request, in batch requests, you need to wait for this request to complete before making the next request

If there is an interval time set, it is necessary to wait for the interval time to end before sending the request.

Multiple crawler application instances

import xCrawl from 'x-crawl'

const myXCrawl1 = xCrawl({
  // options
})

const myXCrawl2 = xCrawl({
  // options
})

Crawl page

Crawl a page via crawlPage() .

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl.crawlPage('https://xxx.com').then((res) => {
  const { browser, page } = res.data

  // Close the browser
  browser.close()
})

browser instance

It is an instance object of Browser. For specific usage, please refer to Browser.

The browser instance is a headless browser without a UI shell. What he does is to bring all modern network platform functions provided by the browser rendering engine to the code.

Note: An event loop will always be generated inside the browser instance, causing the file not to be terminated. If you want to stop, you can execute browser.close() to close it. Do not call crawlPage or page if you need to use it later. Because when you modify the properties of the browser instance, it will affect the browser instance inside the crawlPage API of the crawler instance, the page instance that returns the result, and the browser instance, because the browser instance is shared within the crawlPage API of the same crawler instance.

page instance

It is an instance object of Page. The instance can also perform interactive operations such as events. For specific usage, please refer to [page](https://pptr.dev /api/puppeteer. page).

The browser instance will retain a reference to the page instance. If it is no longer used in the future, the page instance needs to be closed by itself, otherwise it will cause a memory leak.

Take Screenshot

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl.crawlPage('https://xxx.com').then(async (res) => {
  const { browser, page } = res.data

  // Get a screenshot of the rendered page
  await page.screenshot({ path: './upload/page.png' })

  console.log('Screen capture is complete')

  browser.close()
})

Crawl interface

Crawl interface data through crawlData() .

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({ intervalTime: { max: 3000, min: 1000 } })

const requestConfigs = [
  'https://xxx.com/xxxx',
  'https://xxx.com/xxxx',
  { url: 'https://xxx.com/xxxx', method: 'POST', data: { name: 'coderhxl' } }
]

myXCrawl.crawlData({ requestConfigs }).then((res) => {
  // deal with
})

Crawl files

Crawl file data via crawlFile() .

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({ intervalTime: { max: 3000, min: 1000 } })

myXCrawl
  .crawlFile({
    requestConfigs: ['https://xxx.com/xxxx', 'https://xxx.com/xxxx'],
    fileConfig: {
      storeDir: './upload' // storage folder
    }
  })
  .then((res) => {
    console.log(res)
  })

Start polling

Start a polling crawl with startPolling() .

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  timeout: 10000,
  intervalTime: { max: 3000, min: 1000 }
})

myXCrawl.startPolling({ h: 2, m: 30 }, async (count, stopPolling) => {
  // will be executed every two and a half hours
  // crawlPage/crawlData/crawlFile
  const res = await myXCrawl.crawlPage('https://xxx.com')
  res.data.page.close()
})

Using crawlPage in polling Note: The purpose of calling page.close() is to prevent the browser instance from retaining references to the page instance. If the current page is no longer used in the future, it needs to be closed by itself, otherwise it will cause a memory leak.

Callback function parameters:

• The count attribute records the current number of polling operations.
• stopPolling is a callback function, calling it can terminate subsequent polling operations.

Config priority

Some general configuration can be set in three places:

• Examples of crawler applications
• Spider API
• request configuration

The priority is: request config > API config > base config

Interval time

The interval time can prevent too much concurrency and avoid too much pressure on the server.

The crawling interval is controlled internally by the instance method, not the entire crawling interval is controlled by the instance.

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl
  .crawlData({
    requestConfigs: ['https://xxx.com/xxxx', 'https://xxx.com/xxxx'],
    intervalTime: { max: 2000, min: 1000 }
  })
  .then((res) => {})

The intervalTime option defaults to undefined . If there is a setting value, it will wait for a period of time before requesting, which can prevent too much concurrency and avoid too much pressure on the server.

• number: The time that must wait before each request is fixed
• Object: Randomly select a value from max and min, which is more anthropomorphic

Note: The first request will not trigger the interval.

Fail retry

Failed retries can be re-requested when timeouts and the like.

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  intervalTime: { max: 3000, min: 1000 }
})

myXCrawl.crawlData({ url: 'https://xxx.com/xxxx', maxRetry: 1 }).then((res) => {})

The maxRetry attribute determines how many times to retry.

Priority queue

A priority queue allows a request to be sent first.

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  intervalTime: { max: 3000, min: 1000 }
})

myXCrawl
  .crawlData([
    { url: 'https://xxx.com/xxxx', priority: 1 },
    { url: 'https://xxx.com/xxxx', priority: 10 },
    { url: 'https://xxx.com/xxxx', priority: 8 }
  ])
  .then((res) => {})

The larger the value of the priority attribute, the higher the priority in the current crawling queue.

About the results

For the result, the result of each request is uniformly wrapped with an object that provides information about the result of the request, such as id, result, success or not, maximum retry, number of retries, error information collected, and so on. Automatically determine whether the return value is wrapped in an array depending on the configuration you choose, and the type fits perfectly in TS.

API

xCrawl

Create a crawler instance via call xCrawl. The request queue is maintained by the instance method itself, not by the instance itself.

Type

• XCrawlBaseConfig
• XCrawlInstance

function xCrawl(baseConfig?: XCrawlBaseConfig): XCrawlInstance

Example

import xCrawl from 'x-crawl'

// xCrawl API
const myXCrawl = xCrawl({
  baseUrl: 'https://xxx.com',
  timeout: 10000,
  intervalTime: { max: 2000, min: 1000 }
})

Note: To avoid repeated creation of instances in subsequent examples, myXCrawl here will be the crawler instance in the crawlPage/crawlData/crawlFile example.

crawlPage

crawlPage is the method of the crawler instance, usually used to crawl page.

Type

• Look at the CrawlPageConfig type
• Look at the CrawlPageSingleRes type
• Look at the CrawlPageRes type

function crawlPage: <T extends CrawlPageConfig>(
  config: T,
  callback?: ((res: CrawlPageSingleRes) => void) | undefined
) => Promise<CrawlPageRes<T>>

Example

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

// crawlPage API
myXCrawl.crawlPage('https://xxx.com/xxxx').then((res) => {
  const { browser, page } = res.data

  // Close the browser
  browser.close()
})

Config

There are 4 types:

• string
• PageRequestConfig
• (string | PageRequestConfig)[]
• CrawlPageConfigObject

1.string

If you just want to simply crawl this page, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl.crawlPage('https://xxx.com/xxxx').then((res) => {})

2. PageRequestConfig

More configuration options of PageRequestConfig can be found in PageRequestConfig .

If you want to crawl this page and need to retry on failure, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl
  .crawlPage({
    url: 'https://xxx.com/xxxx',
    proxy: 'xxx',
    maxRetry: 1
  })
  .then((res) => {})

3.(string | PageRequestConfig)[]

More configuration options of PageRequestConfig can be found in PageRequestConfig .

If you want to crawl multiple pages, and some pages need to fail and retry, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl
  .crawlPage(['https://xxx.com/xxxx', { url: 'https://xxx.com/xxxx', maxRetry: 2 }])
  .then((res) => {})

4. CrawlPageConfigObject

For more configuration options of CrawlPageConfigObject, please refer to CrawlPageConfigObject .

If you want to crawl multiple pages, and the request configuration (proxy, cookies, retry, etc.) does not want to be written repeatedly, if you need an interval, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl.crawlPage({
   requestConfigs: [
     'https://xxx.com/xxxx',
     { url: 'https://xxx.com/xxxx', maxRetry: 6 }
   ],
   intervalTime: { max: 3000, min: 1000 },
   cookies: 'xxx',
   maxRetry: 1
}).then((res) => {})

It can be selected according to the actual situation.

crawlData

crawlData is the method of the crawler instance, which is usually used to crawl APIs to obtain JSON data and so on.

Type

• Look at the CrawlDataConfig type
• Look at the CrawlDataSingleRes type
• Look at the CrawlDataRes type

function crawlData<D = any, T extends CrawlDataConfig = CrawlDataConfig>(
  config: T,
  callback?: ((res: CrawlDataSingleRes<D>) => void) | undefined
) => Promise<CrawlDataRes<D, T>>

Example

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  timeout: 10000,
  intervalTime: { max: 2000, min: 1000 }
})

myXCrawl
  .crawlData({
    requestConfigs: ['https://xxx.com/xxxx', 'https://xxx.com/xxxx'],
    intervalTime: { max: 3000, min: 1000 },
    cookies: 'xxx',
    maxRetry: 1
  })
  .then((res) => {
    console.log(res)
  })

Config

There are 4 types:

• string
• DataRequestConfig
• (string | DataRequestConfig)[]
• CrawlDataConfigObject

1.string

If you just want to simply crawl the data, and the interface is GET, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl.crawlData('https://xxx.com/xxxx').then((res) => {})

2. DataRequestConfig

More configuration options of DataRequestConfig can be found in DataRequestConfig .

If you want to crawl this data and need to retry on failure, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl
  .crawlData({
    url: 'https://xxx.com/xxxx',
    proxy: 'xxx',
    maxRetry: 1
  })
  .then((res) => {})

3.(string | DataRequestConfig)[]

More configuration options of DataRequestConfig can be found in DataRequestConfig .

If you want to crawl multiple data, and some data needs to fail and retry, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl
  .crawlPage(['https://xxx.com/xxxx', { url: 'https://xxx.com/xxxx', maxRetry: 2 }])
  .then((res) => {})

4. CrawlDataConfigObject

For more configuration options of CrawlPageConfigObject, please refer to CrawlPageConfigObject .

If you want to crawl multiple data, and the request configuration (proxy, cookies, retry, etc.) does not want to be written repeatedly, if you need an interval, you can try this writing method:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl.crawlData({
   requestConfigs: [
     'https://xxx.com/xxxx',
     { url: 'https://xxx.com/xxxx', maxRetry: 6 }
   ],
   intervalTime: { max: 3000, min: 1000 },
   cookies: 'xxx',
   maxRetry: 1
}).then((res) => {})

It can be selected according to the actual situation.

crawlFile

crawlFile is the method of the crawler instance, which is usually used to crawl files, such as pictures, pdf files, etc.

Type

• Look at the CrawlFileConfig type
• Look at the CrawlFileSingleRes type
• Look at the CrawlFileRes type

function crawlFile<T extends CrawlFileConfig>(
  config: T,
  callback?: ((res: CrawlFileSingleRes) => void) | undefined
) => Promise<CrawlFileRes<T>>

Example

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  timeout: 10000,
  intervalTime: { max: 2000, min: 1000 }
})

// crawlFile API
myXCrawl
  .crawlFile({
    requestConfigs: ['https://xxx.com/xxxx', 'https://xxx.com/xxxx'],
    storeDir: './upload',
    intervalTime: { max: 3000, min: 1000 },
    maxRetry: 1
  })
  .then((res) => {})

Config

There are 3 types:

• FileRequestConfig

• FileRequestConfig[]

• CrawlFileConfigObject

1. FileRequestConfig

More configuration options of FileRequestConfig can be found in FileRequestConfig .

If you want to crawl this file and need to retry on failure, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl
  .crawlFile({
    url: 'https://xxx.com/xxxx',
    proxy: 'xxx',
    maxRetry: 1,
    storeDir: './upload',
    fileName: 'xxx'
  })
  .then((res) => {})

2. FileRequestConfig[]

More configuration options of FileRequestConfig can be found in FileRequestConfig .

If you want to crawl multiple files, and some data needs to be retried after failure, you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl
  .crawlFile([
    { url: 'https://xxx.com/xxxx', storeDir: './upload' },
    { url: 'https://xxx.com/xxxx', storeDir: './upload', maxRetry: 2 }
  ])
  .then((res) => {})

3. CrawlFileConfigObject

For more configuration options of CrawlFileConfigObject, please refer to CrawlFileConfigObject .

If you want to crawl multiple data, and the request configuration (storeDir, proxy, retry, etc.) does not want to be written repeatedly, and you need interval time, etc., you can try this way of writing:

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl()

myXCrawl.crawlFile({
   requestConfigs: [
     'https://xxx.com/xxxx',
     { url: 'https://xxx.com/xxxx', storeDir: './upload/xxx' }
   ],
   storeDir: './upload',
   intervalTime: { max: 3000, min: 1000 },
   maxRetry: 1
}).then((res) => {})

It can be selected according to the actual situation.

startPolling

crawlPolling is a method of the crawler instance, typically used to perform polling operations, such as getting news every once in a while.

Type

• Look at the StartPollingConfig type

function startPolling(
  config: StartPollingConfig,
  callback: (count: number, stopPolling: () => void) => void
): void

Example

import xCrawl from 'x-crawl'

const myXCrawl = xCrawl({
  timeout: 10000,
  intervalTime: { max: 2000, min: 1000 }
})

// startPolling API
myXCrawl.startPolling({ h: 2, m: 30 }, (count, stopPolling) => {
  // will be executed every two and a half hours
  // crawlPage/crawlData/crawlFile
})

Types

API Config

IntervalTime

export type IntervalTime = number | { max: number; min?: number }

Method

export type Method =
  | 'get'
  | 'GET'
  | 'delete'
  | 'DELETE'
  | 'head'
  | 'HEAD'
  | 'options'
  | 'OPTIONS'
  | 'post'
  | 'POST'
  | 'put'
  | 'PUT'
  | 'patch'
  | 'PATCH'
  | 'purge'
  | 'PURGE'
  | 'link'
  | 'LINK'
  | 'unlink'
  | 'UNLINK'

PageRequestConfigCookies

export type PageRequestConfigCookies =
  | string
  | Protocol.Network.CookieParam
  | Protocol.Network.CookieParam[]

PageRequestConfig

export interface PageRequestConfig {
  url: string
  headers?: AnyObject
  timeout?: number
  proxy?: string
  cookies?: PageRequestConfigCookies
  maxRetry?: number
  priority?: number
}

DataRequestConfig

export interface DataRequestConfig {
  url: string
  method?: Method
  headers?: AnyObject
  params?: AnyObject
  data?: any
  timeout?: number
  proxy?: string
  maxRetry?: number
  priority?: number
}

FileRequestConfig

export interface FileRequestConfig {
  url: string
  headers?: AnyObject
  timeout?: number
  proxy?: string
  maxRetry?: number
  priority?: number
  storeDir?: string
  fileName?: string
  extension?: string
}

CrawlPageConfigObject

export interface CrawlPageConfigObject {
  requestConfigs: (string | PageRequestConfig)[]
  proxy?: string
  timeout?: number
  cookies?: PageRequestConfigCookies
  intervalTime?: IntervalTime
  maxRetry?: number
}

CrawlDataConfigObject

export interface CrawlDataConfigObject {
  requestConfigs: (string | DataRequestConfig)[]
  proxy?: string
  timeout?: number
  intervalTime?: IntervalTime
  maxRetry?: number
}

CrawlFileConfigObject

export interface CrawlFileConfigObject {
  requestConfigs: (string | FileRequestConfig)[]
  proxy?: string
  timeout?: number
  intervalTime?: IntervalTime
  maxRetry?: number
  fileConfig?: {
    storeDir?: string
    extension?: string
    beforeSave?: (info: {
      id: number
      fileName: string
      filePath: string
      data: Buffer
    }) => Buffer | void
  }
}

XCrawlBaseConfig

export interface XCrawlBaseConfig {
  baseUrl?: string
  timeout?: number
  intervalTime?: IntervalTime
  mode?: 'async' | 'sync'
  proxy?: string
  maxRetry?: number
}

CrawlPageConfig

export type CrawlPageConfig =
  | string
  | PageRequestConfig
  | (string | PageRequestConfig)[]
  | CrawlPageConfigObject

CrawlDataConfig

export type CrawlDataConfig =
  | string
  | DataRequestConfig
  | (string | DataRequestConfig)[]
  | CrawlDataConfigObject

CrawlFileConfig

export type CrawlFileConfig = FileRequestConfig | FileRequestConfig[] | CrawlFileConfigObject

StartPollingConfig

export interface StartPollingConfig {
  d?: number
  h?: number
  m?: number
}

API Result

XCrawlInstance

export interface XCrawlInstance {
  crawlPage: <T extends CrawlPageConfig>(
    config: T,
    callback?: ((res: CrawlPageSingleRes) => void) | undefined
  ) => Promise<CrawlPageRes<T>>

  crawlData: <D = any, T extends CrawlDataConfig = CrawlDataConfig>(
    config: T,
    callback?: ((res: CrawlDataSingleRes<D>) => void) | undefined
  ) => Promise<CrawlDataRes<D, T>>

  crawlFile: <T extends CrawlFileConfig>(
    config: T,
    callback?: ((res: CrawlFileSingleRes) => void) | undefined
  ) => Promise<CrawlFileRes<T>>

  startPolling: (
    config: StartPollingConfig,
    callback: (count: number, stopPolling: () => void) => void
  ) => void
}

CrawlCommonRes

export interface CrawlCommonRes {
  id: number
  isSuccess: boolean
  maxRetry: number
  crawlCount: number
  retryCount: number
  errorQueue: Error[]
}

CrawlPageSingleRes

export interface CrawlPageSingleRes extends CrawlCommonRes {
  data: {
    browser: Browser
    response: HTTPResponse | null
    page: Page
  }
}

CrawlDataSingleRes

export interface CrawlDataSingleRes<D> extends CrawlCommonRes {
  data: {
    statusCode: number | undefined
    headers: IncomingHttpHeaders
    data: D
  } | null
}

CrawlFileSingleRes

export interface CrawlFileSingleRes extends CrawlCommonRes {
  data: {
    statusCode: number | undefined
    headers: IncomingHttpHeaders
    data: {
      isSuccess: boolean
      fileName: string
      fileExtension: string
      mimeType: string
      size: number
      filePath: string
    }
  } | null
}

CrawlPageRes

export type CrawlPageRes<R extends CrawlPageConfig> = R extends
  | (string | PageRequestConfig)[]
  | CrawlPageConfigObject
  ? CrawlPageSingleRes[]
  : CrawlPageSingleRes

CrawlDataRes

export type CrawlDataRes<D, R extends CrawlDataConfig> = R extends
  | (string | DataRequestConfig)[]
  | CrawlDataConfigObject
  ? CrawlDataSingleRes<D>[]
  : CrawlDataSingleRes<D>

CrawlFileRes

export type CrawlFileRes<R extends CrawlFileConfig> = R extends
  | FileRequestConfig[]
  | CrawlFileConfigObject
  ? CrawlFileSingleRes[]
  : CrawlFileSingleRes

API Other

AnyObject

export interface AnyObject extends Object {
  [key: string | number | symbol]: any
}

More

If you have problems, needs, good suggestions please raise Issues in https://github.com/coder-hxl/x-crawl/issues.

Changelog

v5.0.0 (2023-04-06)

🚨 Breaking Changes

• For configuration, major changes have been made to each crawling API configuration, and the same API supports more crawling configuration methods, each of which has its own significance.
• For the result, the result of each request will be wrapped in an object, which provides information about the result of this request, such as: id, result, success, maximum retry, number of retries, collected error information, etc. . Automatically determine whether the return value is wrapped in an array according to the configuration method you choose, and the type is perfectly matched in TS.
• For obtaining results through the callback function, the callback is no longer executed after a single request is completed like the v4.x version, but will be executed sequentially after the crawling is completed, which will not block subsequent crawling.

🚀 Features

• Added a retry mechanism, which can be set for all crawling requests, for a single crawling request, and for a single request.
• A new priority queue is added to use priority crawling according to the priority of a single request.
• For more configurations that may be reused, you can set the baseConfig settings passed in when requesting configuration, API crawling configuration, and generating crawler instances, such as: timeout, proxy, intervalTime, etc., and the weight is: requestConfig > APIConfig > baseConfig.
• For crawlFile API, file path, name, suffix and other information can be set individually for each file. Added the beforeSave life cycle function before saving the file. You can get the file data of the Buffer type, and you can perform operations such as compression on the data in the callback. The returned new Buffer data will replace the original data and write it into the file.
• Update the output of crawling on the console, and collect the error information generated by crawling into an error queue. After the crawling is completed, you can get the error message queue through the return value.

---

🚨 重大改变

• 对于配置，每个爬取 API 配置发生重大改变，同一个 API 支持更多爬取配置方式，每种方式都有其存在的意义。
• 对于结果，每个请求的结果将统一使用对象包裹着，该对象提供了关于这次请求结果的信息，比如：id、结果、是否成功、最大重试、重试次数、收集到错误信息等。自动根据你选用的配置方式决定返回值是否包裹在一个数组中，在 TS 中类型完美适配。
• 对于通过回调函数方式获取结果，该回调不再像 v4.x 版本在单个请求完成后执行，而是将会在爬取完成后按顺序执行，这将不会阻塞后续的爬取。

🚀 特征

• 新增失败重试，可针对所有爬取的请求设置，针对单次爬取的请求设置，针对单个请求设置进行失败重试。
• 新增优先队列，根据单个请求的优先级使用优先爬取。
• 对更多可能复用的配置可以在请求配置、API 爬取配置、生成爬虫实例时传入的 baseConfig 设置，比如：timeout、proxy、intervalTime 等，权重为：requestConfig > APIConfig > baseConfig。
• 对 crawlFile API 可单独为每个文件设置文件路径、名字、后缀等信息。新增 beforeSave 文件保存前生命周期函数，可拿到 Buffer 类型的文件数据，可在回调内对数据进行压缩等操作，返回新的 Buffer 数据会替代原先的数据写入文件中。
• 对爬取在控制台的输出更新，对爬取产生的报错信息分别收集到一个的错误队列中，爬取完成后可通过返回值拿到该错误信息队列。