rss_parser_api

Endpoints

GET /items

Returns the RSS/Atom feed's items parsed as JSON.

Each item will include its enclosure's MD5 checksum as a checksum property.

Parameters

• feed {url} (required): RSS/Atom URL to parse
• verbose {boolean} (default: true): whether items' whole information should be included
• pretty {boolean} (default: true): whether to indent the JSON response
• checksum {boolean} (default: true): whether checksums should be included. Specifying false significantly speed up the request.

Success response

{
  "data": [
    {
      "title": "24. Podcast title",
      "description": "Description of this podcast",
      "checksum": "95ceab9bcd3c1c27b268a6375d9a8032",
      ...
    },
    ...
  ]
}

Error responses

The following type are available:

• INPUT_VALIDATION (400): wrong parameters. This includes supplying a document containing a URL to a feed or media file that does not exist or has syntax errors.
• FEED (400): feed is not valid XML
• NOT_FOUND (404): wrong request URL
• METHOD_NOT_ALLOWED (405): unsupported HTTP method
• UNKNOWN (500): an unknown server-side error occurred

GET /id3

Returns the MP3 file's ID3 tags.

Parameters

• url {url} (required): URL to an MP3 file
• pretty {boolean} (default: true): whether to indent the JSON response

Success response

{
  "data": {
    "TIT2": {
      "id": "TIT2",
      "size": 14,
      "description": "Title/songname/content description",
      "data": "24. Summer hit"
    },
    ...
  ]
}

Error responses

The same type as GET /items are available except FEED.

Starting the server

Run npm install then npm start.

To run in dev mode, run npm run watch.

To perform linting and testing, run npm test.

Configuration

The following configuration properties can be set using environment variables:

• HOST (default: localhost)
• PORT (default: 80)
• NODE_ENV (default: development)

Parameters design

We allow REST parameters both as:

• query variables, e.g. ?feed=URL, because it is the most common/expected practice and it is easier for debugging. Values must be URI encoded.
• HTTP headers, e.g. X-Rss-Parser-Feed: URL, because it is the most semantically correct. According to W3C TAG specs, URLs should be used only for resource identification, not for query logic. Protocol headers should be used for that instead.

Features

• multiple syndication formats support
• response streaming
• performance
• logging
• proper error handling
• basic performance monitoring
• follows W3C and IETF guidelines on REST design
• security
• microservice-ready
• universal JavaScript

Syndication formats

The following formats are supported: RSS 2.0, RSS 1.0, RSS 0.90, Atom 1.0, Atom 0.3, RDF Feed.

Moreover MRSS, itunes and Dublin core properties are supported.

Response streaming

Calculating checksums with GET /items is slow because each media file must be downloaded first. To mitigate this:

• the response is streamed with HTTP/1.1 chunked transfer encoding.
• media files are downloaded in parallel as soon as their URI is available. However the order of the items in the response is properly kept.

Performance

Also response body are compressed with deflate and gzip.

Furthermore caching is being performed:

• between the server and the URLs being fetched
• between the client and the server as both conditional and unconditional HTTP caching

Logging

Logs are printed on console as JSON, one message per line. This is to dissociate logs creation from logs transport.

The three following types of logs are performed.

start

When the server starts.

{
  "type": "start",
  "level": "log",
  "message": "Server listening at http://localhost:5001"
}

request

When a client makes an HTTP request, successful or not.

{
  "type": "request",
  "level": "log",
  "statusCode": 200,
  "method": "GET",
  "path": "/id3",
  "query": { "url":"file.mp3" },
  "ip": "127.0.0.1"
}

error

When a client makes an HTTP request that fails.

{
  "type": "error",
  "level": "error",
  "error": { ... }
}

The error property has the same signature as the error response

Error handling

Error responses follow the RFC 7807 "problem details":

{
  "error": {
    "status": 404,
    "type": "NOT_FOUND",
    "description": "Route '/id33' does not exist",
    "instance": "http://localhost:5001/id33",
    "details": "Error: Route '/id33' does not exist\n    at ..."
  }
}

Stack traces are only included when NODE_ENV is development.

Performance monitoring

The X-Response-Time HTTP header is set on each response.

REST design

The method GET is used, as opposed to POST, since the calls are safe.

Semantic HTTP response headers are included:

• Last-Modified based on the feed's <lastPublishedDate>, <modified> or <updated>
• Link: <URI>; rel="alternate"; type="application/rss+xml"

Security

Basic security is provided by helmet.

Microservice

This project follows the 12-factor app principles.

Universal JavaScript

Most dependencies work on the browser as well, so the core functions of this microservice can easily be ported client-side.

Troubleshooting

Please use latest Node version.

Missing features

The following features could be added (in priority order):

• pagination
• improving the security: HTTPS, authorization, more server limits, etc.
• HTTP/2
• DevOps, Continuous Integration/Deployment
• using a distributed key-value store for server-side caching
• filtering, e.g. with a filter parameter
• sorting, e.g. with an order parameter
• request timeout
• server push (WebSub) or long polling
• improving the API documentation, e.g. with a separate website
• improving the performance monitoring
• versioning
• fetching more fields in the feed items, e.g. their thumbnails
• using a streaming-friendly response format like JSON text sequences, jsonlines or ndjson
• JSON Feed support
• content-negotiation, e.g. letting clients specify the charset and format
• support for other RPCs, e.g. GraphQL or JSON-RPC
• support for other protocols, e.g. WebSocket
• using the <ttl> RSS element for HTTP unconditional caching
• HATEOAS