x-crawl

x-crawl

English | 简体中文

XCrawl is a Nodejs multifunctional crawler library. Crawl HTML, JSON, file resources, etc. through simple configuration.

Install

Take NPM as an example:

npm install x-crawl

example

Get the title of https://docs.github.com/zh/get-started as an example:

// Import module ES/CJS
import XCrawl from 'x-crawl'

// Create a crawler instance
const docsXCrawl = new XCrawl({
  baseUrl: 'https://docs.github.com',
  timeout: 10000,
  intervalTime: { max: 2000, min: 1000 }
})

// Call fetchHTML API to crawl
docsXCrawl.fetchHTML('/zh/get-started').then((jsdom) => {
  console.log(jsdom.window.document.querySelector('title')?.textContent)
})

Key concept

XCrawl

Create a crawler instance via new XCrawl.

• Type

class XCrawl {
  private readonly baseConfig
  constructor(baseConfig?: IXCrawlBaseConifg)
  fetchHTML(config: string | IFetchHTMLConfig): Promise<JSDOM>
  fetchData<T = any>(config: IFetchDataConfig): Promise<IFetchCommon<T>>
  fetchFile(config: IFetchFileConfig): Promise<IFetchCommon<IFileInfo>>
}

• Example

myXCrawl is the crawler instance of the following example.

const myXCrawl = new XCrawl({
  baseUrl: 'https://xxx.com',
  timeout: 10000,
  // The interval of the next request, multiple requests are valid
  intervalTime: {
    max: 2000,
    min: 1000
  }
})

fetchHTML

fetchHTML is the method of the above myXCrawl instance, usually used to crawl HTML.

• Type

function fetchHTML(config: string | IFetchHTMLConfig): Promise<JSDOM>

• Example

myXCrawl.fetchHTML('/xxx').then((jsdom) => {
  console.log(jsdom.window.document.querySelector('title')?.textContent)
})

fetchData

fetchData is the method of the above myXCrawl instance, which is usually used to crawl APIs to obtain JSON data and so on.

• Type

function fetchData<T = any>(config: IFetchDataConfig): Promise<IFetchCommon<T>>

• Example

const requestConifg = [
  { url: '/xxxx', method: 'GET' },
  { url: '/xxxx', method: 'GET' },
  { url: '/xxxx', method: 'GET' }
]

myXCrawl.fetchData({ 
  requestConifg, // Request configuration, can be IRequestConfig | IRequestConfig[]
  intervalTime: 800 // Interval between next requests, multiple requests are valid
}).then(res => {
  console.log(res)
})

fetchFile

fetchFile is the method of the above myXCrawl instance, which is usually used to crawl files, such as pictures, pdf files, etc.

• Type

function fetchFile(config: IFetchFileConfig): Promise<IFetchCommon<IFileInfo>>

• Example

const requestConifg = [
  { url: '/xxxx' },
  { url: '/xxxx' },
  { url: '/xxxx' }
]

myXCrawl.fetchFile({
  requestConifg,
  fileConfig: {
    storeDir: path.resolve(__dirname, './upload') // storage folder
  }
}).then(fileInfos => {
  console.log(fileInfos)
})

Types

• IAnyObject

interface IAnyObject extends Object {
  [key: string | number | symbol]: any
}

• IMethod

type IMethod = 'get' | 'GET' | 'delete' | 'DELETE' | 'head' | 'HEAD' | 'options' | 'OPTIONS' | 'post' | 'POST' | 'put' | 'PUT' | 'patch' | 'PATCH' | 'purge' | 'PURGE' | 'link' | 'LINK' | 'unlink' | 'UNLINK'

• IRequestConfig

interface IRequestConfig {
  url: string
  method?: IMethod
  headers?: IAnyObject
  params?: IAnyObject
  data?: any
  timeout?: number
}

• IIntervalTime

type IIntervalTime = number | {
  max: number
  min?: number
}

• IFetchBaseConifg

interface IFetchBaseConifg {
  requestConifg: IRequestConfig | IRequestConfig[]
  intervalTime?: IIntervalTime
}

• IFetchCommon

type IFetchCommon<T> = {
  id: number
  statusCode: number | undefined
  headers: IncomingHttpHeaders // node:http type
  data: T
}[]

• IFileInfo

IFileInfo {
  fileName: string
  mimeType: string
  size: number
  filePath: string
}

• IXCrawlBaseConifg

interface IXCrawlBaseConifg {
  baseUrl?: string
  timeout?: number
  intervalTime?: IIntervalTime
}

• IFetchHTMLConfig

interface IFetchHTMLConfig extends IRequestConfig {}

• IFetchDataConfig

interface IFetchDataConfig extends IFetchBaseConifg {
}

• IFetchFileConfig

interface IFetchFileConfig extends IFetchBaseConifg {
  fileConfig: {
    storeDir: string
  }
}

More

If you have any questions or needs , please submit Issues in https://github.com/coder-hxl/x-crawl/issues .

---

x-crawl

English | 简体中文

XCrawl 是 Nodejs 多功能爬虫库。只需简单的配置即可抓取 HTML 、JSON、文件资源等等。

安装

以 NPM 为例:

npm install x-crawl

示例

获取 https://docs.github.com/zh/get-started 的标题为例:

// 导入模块 ES/CJS
import XCrawl from 'x-crawl'

// 创建一个爬虫实例
const docsXCrawl = new XCrawl({
  baseUrl: 'https://docs.github.com',
  timeout: 10000,
  intervalTime: { max: 2000, min: 1000 }
})

// 调用 fetchHTML API 爬取
docsXCrawl.fetchHTML('/zh/get-started').then((jsdom) => {
  console.log(jsdom.window.document.querySelector('title')?.textContent)
})

核心概念

XCrawl

通过 new XCrawl 创建一个爬虫实例。

• 类型

class XCrawl {
  private readonly baseConfig
  constructor(baseConfig?: IXCrawlBaseConifg)
  fetchHTML(config: string | IFetchHTMLConfig): Promise<JSDOM>
  fetchData<T = any>(config: IFetchDataConfig): Promise<IFetchCommon<T>>
  fetchFile(config: IFetchFileConfig): Promise<IFetchCommon<IFileInfo>>
}

• 示例

myXCrawl 为后面示例的爬虫实例。

const myXCrawl = new XCrawl({
  baseUrl: 'https://xxx.com',
  timeout: 10000,
  // 下次请求的间隔时间, 多个请求才有效
  intervalTime: {
    max: 2000,
    min: 1000
  }
})

fetchData

fetch 是上面 myXCrawl 实例的方法，通常用于爬取 API ，可获取 JSON 数据等等。

• 类型

function fetchData<T = any>(config: IFetchDataConfig): Promise<IFetchCommon<T>>

• 示例

const requestConifg = [
  { url: '/xxxx', method: 'GET' },
  { url: '/xxxx', method: 'GET' },
  { url: '/xxxx', method: 'GET' }
]

myXCrawl.fetchData({ 
  requestConifg, // 请求配置, 可以是 IRequestConfig | IRequestConfig[]
  intervalTime: 800 // 下次请求的间隔时间, 多个请求才有效
}).then(res => {
  console.log(res)
})

fetchHTML

fetchHTML 是上面 myXCrawl 实例的方法，通常用于爬取 HTML 。

• 类型

function fetchHTML(config: string | IFetchHTMLConfig): Promise<JSDOM>

• 示例

myXCrawl.fetchHTML('/xxx').then((jsdom) => {
  console.log(jsdom.window.document.querySelector('title')?.textContent)
})

fetchFile

fetchFile 是上面 myXCrawl 实例的方法，通常用于爬取文件，可获取图片、pdf 文件等等。

• 类型

function fetchFile(config: IFetchFileConfig): Promise<IFetchCommon<IFileInfo>>

• 示例

const requestConifg = [
  { url: '/xxxx' },
  { url: '/xxxx' },
  { url: '/xxxx' }
]

myXCrawl.fetchFile({
  requestConifg,
  fileConfig: {
    storeDir: path.resolve(__dirname, './upload') // 存放文件夹
  }
}).then(fileInfos => {
  console.log(fileInfos)
})

类型

• IAnyObject

interface IAnyObject extends Object {
  [key: string | number | symbol]: any
}

• IMethod

type IMethod = 'get' | 'GET' | 'delete' | 'DELETE' | 'head' | 'HEAD' | 'options' | 'OPTIONS' | 'post' | 'POST' | 'put' | 'PUT' | 'patch' | 'PATCH' | 'purge' | 'PURGE' | 'link' | 'LINK' | 'unlink' | 'UNLINK'

• IRequestConfig

interface IRequestConfig {
  url: string
  method?: IMethod
  headers?: IAnyObject
  params?: IAnyObject
  data?: any
  timeout?: number
}

• IIntervalTime

type IIntervalTime = number | {
  max: number
  min?: number
}

• IFetchBaseConifg

interface IFetchBaseConifg {
  requestConifg: IRequestConfig | IRequestConfig[]
  intervalTime?: IIntervalTime
}

• IFetchCommon

type IFetchCommon<T> = {
  id: number
  statusCode: number | undefined
  headers: IncomingHttpHeaders // node:http type
  data: T
}[]

• IFileInfo

interface IFileInfo {
  fileName: string
  mimeType: string
  size: number
  filePath: string
}

• IXCrawlBaseConifg

interface IXCrawlBaseConifg {
  baseUrl?: string
  timeout?: number
  intervalTime?: IIntervalTime
}

• IFetchHTMLConfig

interface IFetchHTMLConfig extends IRequestConfig {}

• IFetchDataConfig

interface IFetchDataConfig extends IFetchBaseConifg {
}

• IFetchFileConfig

interface IFetchFileConfig extends IFetchBaseConifg {
  fileConfig: {
    storeDir: string
  }
}

更多

如有 问题 或 需求 请在 https://github.com/coder-hxl/x-crawl/issues 中提 Issues 。

Changelog

v0.1.1 (2023-01-31)

🚀 Features

• fetchHTML API parameter can be Object type