post-feed-reader

post-feed-reader

A library to fetch news, blog or podcast posts from any site. It works by auto-discovering a post source, which can be an RSS/Atom/JSON feed or the Wordpress REST API, and then fetches and parses the list of posts.

It's originally meant for NodeJS, but as it is built on Isomorphic Javascript, it can work on browsers if the website allows cross-origin requests.

Originally built for apps that need to list the posts with their own UI, but don't actually manage the blog and need automatic fallbacks when the blog technology does change.

Features

• Simple: Two-liner usage. Just fetches and parses posts.
• Supports multiple sources: Wordpress REST API, RSS 2.0.1, RSS 2.0, RSS 1.0, RSS 0.91, Atom 1.0 and JSON Feed 1.1.
• Auto-discovery: Give any site URL and the library will try to find the data automatically.

Getting Started

Install it with NPM or Yarn:

npm install post-feed-reader

yarn add post-feed-reader

You first need to discover the post source, which will be a URL to the RSS/Atom/JSON Feed or the Wordpress REST API.

Then you can pass the discovered source to the getPostList, which will fetch and parse it.

import { discoverPostSource, getPostList } from 'post-feed-reader';

// Looks for metadata pointing to the Wordpress REST API or Atom/RSS Feeds
const source = await discoverPostSource('https://www.nytimes.com');

// Retrieves the posts from the given source
const posts = await getPostList(source);

console.log(posts);

Simple enough, eh?

Output

See an example of the result based on the Mozilla blog.

Options

const source = await discoverPostSource('https://techcrunch.com', {
  // Custom axios instance
  axios: axios.create(...),

  // Whether it will prioritize feeds over the wordpress api
  preferFeeds: false,

  // Custom data source filtering
  canUseSource: (source: DiscoveredSource) => true,

  // Whether it will try to guess wordpress api and feed urls if auto-discovery doesn't work
  tryToGuessPaths: false,
  
  // The paths that it will try to guess for both the Wordpress API or the RSS/Atom/JSON feed
  wpApiPaths: ['./wp-json', '?rest_route=/'],
  feedPaths: ['./feed', './atom', './rss', './feed.json', './feed.xml', '?feed=atom'],
});

const posts = await getPostList(source, {
  // Custom axios instance
  axios: axios.create(...),

  // Whether missing plain text contents will be filled automatically from html contents
  fillTextContents: false,

  // Wordpress REST API only options
  wordpress: {
    // Whether it will include author, taxonomy and media data from the wordpress api
    includeEmbedded: true,

    // The amount of items to return
    limit: 10,

    // The page number
    page: 1,

    // The search string filter
    search: '',

    // The author id filter
    authors: [...],

    // The category id filter
    categories: [...],

    // The tag id filter
    tags: [...],

    // Any additional querystring parameter for the wordpress api you may want to include
    additionalParams: { ... },
  },
});

Skip the auto-discovery

If you already have an Atom/RSS/JSON Feed or the Wordpress REST API url in hands, you can fetch the posts directly:

// RSS, Atom or JSON Feed
const feedPosts = await getFeedPostList('https://news.google.com/atom');

// Wordpress API
const wpApiPosts = await getWordpressPostList('https://blog.mozilla.org/en/wp-json/');

Why support the Wordpress REST API, isn't RSS enough?

RSS is indeed the most widely feed format used on the web, but not only it lacks data, but the specification is a mess with many vague to implementation properties, meaning how the information is formatted differs from feed to feed. For instance, the description can be the full post as HTML, or just an excerpt, or in plain text, or even just a HTML link to the post page.

Atom's specification is way more rigid and robust, which makes relying on the data trustworthier. It's definitely the way to go in the topic of feeds. But it still lacks some properties that can only be fetched through the Wordpress REST API.

The Wordpress REST API also has the following benefits:

• Filtering by category, tag or author
• Searching
• Pagination
• Featured media
• Author profile

The JSON Feed format is also just as good as Atom format, but very few websites produce it.

How does the auto-discovery works?

1. Fetches the site's main page
2. Looks for Wordpress Link headers
3. Looks for RSS, Atom and JSON Feed <link> metatags
4. If tryToGuessPaths is set to true, it will look for the paths to try to find a feed or the WP API.