@extractus/feed-extractor

feed-extractor

To read & normalize RSS/ATOM/JSON feed data.

Intro

feed-extractor is a part of tool sets for content builder:

• feed-extractor: extract & normalize RSS/ATOM/JSON feed
• article-extractor: extract main article from given URL
• oembed-extractor: extract oEmbed data from supported providers

You can use one or combination of these tools to build news sites, create automated content systems for marketing campaign or gather dataset for NLP projects...

Attention

feed-reader has been renamed to @extractus/feed-extractor since v6.1.4

Demo

• Give it a try!
• Example FaaS

Install & Usage

Node.js

npm i @extractus/feed-extractor

# pnpm
pnpm i @extractus/feed-extractor

# yarn
yarn add @extractus/feed-extractor

// es6 module
import { read } from '@extractus/feed-extractor'

// CommonJS
const { read } = require('@extractus/feed-extractor')

// or specify exactly path to CommonJS variant
const { read } = require('@extractus/feed-extractor/dist/cjs/feed-extractor.js')

Deno

// deno < 1.28
import { read } from 'https://esm.sh/@extractus/feed-extractor'

// deno > 1.28
import { read } from 'npm:@extractus/feed-extractor'

Browser

import { read } from 'https://unpkg.com/@extractus/feed-extractor@latest/dist/feed-extractor.esm.js'

Please check the examples for reference.

Deta cloud

For Deta devs please refer the source code and guideline here or simply click the button below.

APIs

read()

Load and extract feed data from given RSS/ATOM/JSON source. Return a Promise object.

Syntax

read(String url)
read(String url, Object options)
read(String url, Object options, Object fetchOptions)

Parameters

url required

URL of a valid feed source

Feed content must be accessible and conform one of the following standards:

• RSS Feed
• ATOM Feed
• JSON Feed

For example:

import { read } from '@extractus/feed-extractor'

const result = await read('https://news.google.com/atom')
console.log(result)

Without any options, the result should have the following structure:

{
  title: String,
  link: String,
  description: String,
  generator: String,
  language: String,
  published: ISO Date String,
  entries: Array[
    {
      title: String,
      link: String,
      description: String,
      published: ISO Datetime String
    },
    // ...
  ]
}

options optional

Object with all or several of the following properties:

• normalization: Boolean, normalize feed data or keep original. Default true.
• useISODateFormat: Boolean, convert datetime to ISO format. Default true.
• descriptionMaxLen: Number, to truncate description. Default 210 (characters).
• xmlParserOptions: Object, used by xml parser, view fast-xml-parser's docs
• getExtraFeedFields: Function, to get more fields from feed data
• getExtraEntryFields: Function, to get more fields from feed entry data

For example:

import { read } from '@extractus/feed-extractor'

await read('https://news.google.com/atom', {
  useISODateFormat: false
})

await read('https://news.google.com/rss', {
  useISODateFormat: false,
  getExtraFeedFields: (feedData) => {
    return {
      subtitle: feedData.subtitle || ''
    }
  },
  getExtraEntryFields: (feedEntry) => {
    const {
      enclosure,
      category
    } = feedEntry
    return {
      enclosure: {
        url: enclosure['@_url'],
        type: enclosure['@_type'],
        length: enclosure['@_length']
      },
      category: isString(category) ? category : {
        text: category['@_text'],
        domain: category['@_domain']
      }
    }
  }
})

fetchOptions optional

You can use this param to set request headers to fetch.

For example:

import { read } from '@extractus/feed-extractor'

const url = 'https://news.google.com/rss'
await read(url, null, {
  headers: {
    'user-agent': 'Opera/9.60 (Windows NT 6.0; U; en) Presto/2.1.1'
  }
})

You can also specify a proxy endpoint to load remote content, instead of fetching directly.

For example:

import { read } from '@extractus/feed-extractor'

const url = 'https://news.google.com/rss'

await read(url, null, {
  headers: {
    'user-agent': 'Opera/9.60 (Windows NT 6.0; U; en) Presto/2.1.1'
  },
  proxy: {
    target: 'https://your-secret-proxy.io/loadXml?url=',
    headers: {
      'Proxy-Authorization': 'Bearer YWxhZGRpbjpvcGVuc2VzYW1l...'
    }
  }
})

Passing requests to proxy is useful while running @extractus/feed-extractor on browser. View examples/browser-feed-reader as reference example.

Quick evaluation

git clone https://github.com/extractus/feed-extractor.git
cd feed-extractor
npm install

node eval.js --url=https://news.google.com/rss --normalization=y --useISODateFormat=y --includeEntryContent=n --includeOptionalElements=n

License

The MIT License (MIT)

---