rock-req

Rock-req.js

Ensure your HTTP requests always reach their destination!

🔥 Why should you need this?

In most existing libraries (2023):

• Managing reliable retries is difficult, tricky with streams and not battle-tested
• Using multiple forward proxies has several benefits like higher availability and increased bandwidth but Intercepting retries to use another Egress controller between two requests is not possible.
• Many request libraries are heavy: node-fetch, superagent, needle, got, axios, request
• Lightweight alternatives are not as light as they claim due to dependencies (simple-get, tiny-req, puny-req, ...)

⚡️ Rock-req solves these problems with only 160 lines of code and zero dependencies

It also supports many features:

• Follows redirects
• Handles gzip/deflate/brotli responses
• Modify defaults
• Extend and create new instances
• Automatically destroy input/output stream on error (pipeline)
• Composable
• Timeouts
• HTTPS / HTTP
• Composes well with npm packages for features like cookies, proxies, form data, & OAuth
• Typescript definition
• Keep 98% of the simple-get API (fork source)

Like NodeJS pipeline, when the callback is called, the request is 100% finished, even with streams.

Install

  npm install rock-req

Usage

All functions accept two or three parameters:

  const rock = require('rock-req')
  rock(optsOrURL [, bodyOrStreamFn], callback)

• optsOrURL can be an object or a the URL
• bodyOrStreamFn can be a buffer/string/object or a function returning an input stream for sending the body of POST/PUT/PATCH/DELETE requests
• callback(err, res, data) called only when everything is finished (even with streams).

GET, HEAD requests

// res is the server response, already consumed by rock-req. The result is in data
rock.get('http://ex.com', (err, res, data) => {
  console.log(res.statusCode) // 200
  console.log(data) // Buffer('server response')
})

Alternative syntax:

rock({ method: 'GET', url: 'http://ex.com' }, function (err, res, data) {} )
// OR
rock.concat({ method: 'GET', url: 'http://ex.com' }, function (err, res, data) {} )

Head requests:

rock.head('http://example.com', (err, res, data) => {})

POST, PUT, PATCH, DELETE requests

Use the second paramater to pass the body:

rock.post('http://ex.com', 'POST body', (err, res, data) => {})

Alternative syntax:

rock({ method: 'POST', url : 'http://ex.com', body : 'POST body' }, function (err, res, data) {} )

JSON request shortcuts

Automatically serialize/deserialize request and response with JSON with getJSON, putJSON, postJSON, deleteJSON, ...

rock.putJSON('http://ex.com', { id : 123 }, (err, res, data) => {
  console.log(data) // already JSON.parsed
})

Alternative syntax:

rock({ method: 'PUT', url: 'http://ex.com', body: { id : 123 }, json: true }, function (err, res, data) {} )

All options:

const opts = {
  url    : 'http://example.com',
  method : 'POST',
  body   : 'this is the POST body',
  headers: {
    'user-agent': 'my cool app'
  }
}
rock(opts, function (err, res, data) {} )

opts can contain any value of NodeJS http.request with rock-req parameters. Here are the most used one:

• maxRedirects <number>overwrite global maximum number of redirects. Defaults to 10
• maxRetry <number> overwrite global maximum number of retries. Defaults to 0
• followRedirects <boolean> do not follow redirects
• body <buffer> | <string> | <object> | <function> body to post
• json <boolean> automatically stringify/parse request/response Default : false
• url <string> the destination URL
• method <string> A string specifying the HTTP request method. Default: 'GET'.
• headers <object> An object containing request headers. Default: 'accept-encoding': 'gzip, deflate, br'
• timeout <number>: A number specifying the socket timeout in milliseconds.
• auth <string> Basic authentication ('user:password') to compute an Authorization header.
• port <number> Port of remote server. Default: defaultPort if set, else 80.
• host <string> A domain name or IP address of the server to issue the request to. Default: 'localhost'.
• hostname <string> Alias for host.
• path <string> Request path. Should include query string if any. E.G. '/index.html?page=12'.
• protocol <string> Protocol to use. Default: 'http:'.
• setHost <boolean>: Specifies whether or not to automatically add the Host header. Defaults to true.
• agent <http.Agent> | <boolean> Controls Agent behavior. Possible values:
  • undefined (default): use http.globalAgent for this host and port.
  • Agent object: explicitly use the passed in Agent.
  • false causes a new Agent with default values to be used.

Input Stream

Rock-req requires that input stream is initialized in a function.

This function is invoked by rock-req for every request retry. If something goes wrong, the old stream is destroyed.

const rock = require('rock-req')
const fs = require('fs')

// opts contains options passed in rock(opts). DO NOT MODIFY IT
function createInputStream(opts) {
  return fs.createReadStream('input.txt');
}

const opts = {
  url : 'http://example.com',
  body: createInputStream
}
rock(opts, function (err, res, data) {})

Alternative syntax:

  rock.post('http://example.com', createInputStream, function (err, res, data) {})
  // or with more object options:
  rock.post(opts, createInputStream, function (err, res, data) {})

Output Stream

Rock-req requires that output stream is initialized in a function. This function is invoked by rock-req for every request retry.

const rock = require('rock-req')
const fs = require('fs')
const { finished } = require('stream')

// opts contains options passed in rock(opts). DO NOT MODIFY IT
// res  if the http response (res.statusCode, ...). DO NOT MODIFY IT and DO NOT CONSUME THE RES STREAM YOURSELF
function createOutputStream(opts, res) {

  const writer = fs.createWriteStream('test_gfg.txt') 
  // Internally, rock-req uses pipeline. If something goes wrong, the stream is destroyed automatically.
  // If you need to do some action (removing temporary files, ...), uses this native NodeJS method:
  const cleanup = finished(writer, (err) => {
    if (err) {
      // clean up things 
    } 
    // When using the finished() method in NodeJS, it's important to be aware that it can leave some event listeners 
    // (specifically, the 'error', 'end', 'finish', and 'close' events) hanging around even after this callback function has been called.
    // This is intentional, as it helps prevent unexpected crashes if an error occurs due to incorrect stream implementations.
    // However, if you don't want these event listeners to stick around after the callback function has been called,
    // you can use the cleanup function that's returned by stream.finished() to remove them. 
    // You'll need to explicitly call this cleanup function within your callback function to ensure that the event listeners get removed properly.
    cleanup();
  });
  // It must return a Writable stream. Otherwise, the request is cancel with an error
  return writer
}

const opts = {
  url    : 'http://example.com',
  output : createOutputStream
}
rock(opts, function (err, res) {})

Retry on failure

By default, rock-req retries with the following errors if maxRetry > 1.

The callback is called when the request succeed or all retries are done

const rock = require('rock-req');

// default values can be overwritten like this:
rock.defaults.retryOnCode = [
  408, /* Request Timeout */
  429, /* Too Many Requests */
  500, /* Internal Server Error */
  502, /* Bad Gateway */
  503, /* Service Unavailable */
  504, /* Gateway Timeout*/
  521, /* Web Server Is Down*/
  522, /* Cloudflare Connection Timed Out */
  524  /* Cloudflare A Timeout Occurred */
];
rock.defaults.retryOnError = [
  'ETIMEDOUT', /* One of the timeout limits was reached. */
  'ECONNRESET', /* The connection was forcibly closed. */
  'EADDRINUSE', /* Could not bind to any free port */
  'ECONNREFUSED', /* The connection was refused by the server. */
  'EPIPE', /* The remote side of the stream being written has been closed. */
  'ENOTFOUND', /* Could not resolve the hostname to an IP address. */
  'ENETUNREACH', /*  No internet connection. */
  'EAI_AGAIN' /* DNS lookup timed out. */
];

const opts = {
  url      : 'http://example.com',
  body     : 'this is the POST body',
  maxRetry : 2 // 0 is the default value (= no retries)
}
rock(opts, function (err, res, data) {} );

Global options

Change default parameters globally (not recommended), or create a new instance with specific paramaters (see below)

rock.defaults = {
  headers       : { 'accept-encoding': 'gzip, deflate, br' },
  maxRedirects  : 10,
  maxRetry      : 0,
  retryDelay    : 100, //ms
  retryOnCode   : [408, 429, 500, 502, 503, 504, 521, 522, 524 ],
  retryOnError  : ['ETIMEDOUT', 'ECONNRESET', 'EADDRINUSE', 'ECONNREFUSED','EPIPE', 'ENOTFOUND', 'ENETUNREACH', 'EAI_AGAIN' ],
  // beforeRequest is called for each request, retry and redirect
  beforeRequest : (opts) => {
    // There options can be overwritted (= parsed opts.url)
    opts.protocol = 'https:' //  or 'http:' 
    opts.hostname = 'google.com';
    opts.port = 443;
    opts.path = '/mypage.html?bla=1#hash';
    opts.auth = '';
    opts.headers = {};
    opts.body = {};
    opts.method = 'POST';
    opts.remainingRetry; 
    opts.remainingRedirects;
    
    // READ-ONLY options (not exhaustive)
    opts.url; // DOT NOT OVERWRITE
    opts.maxRetry;
    opts.maxRedirects;
    opts.prevError; // error of previous request on retry
    opts.prevStatusCode; // HTTP status code of previous request on retry/redirect

    // opts must be returned
    return opts;
  },
}

Extend and intercept retries

Create a new instance with specific parameter instead of modifying rock.defaults

By default, this new instance inherits values of the instance source if options are not overwritten. Headers are merged. Then only the first level of the options object is merged (no deep travelling in sub-objects or arrays).

Here is a basic example of beforeRequest interceptor to use HAProxy as a forward proxy.

beforeRequest is always called on each redirect/retry.

• on redirect, opts.url (and hostname, port, protocol, path) is updated to the new location. opts.url is null if it is a relative redirect.
• on retry, opts.url (and hostname, port, protocol, path) have the same value as they did when the rock-req was initially called.

const myInstance = rock.extend({
  beforeRequest: (opts) => {
    const { hostname, port, protocol, path } = opts;
    opts.protocol = 'http:';
    opts.hostname = '10.0.0.1';
    opts.port = 80;
    opts.path = `${hostname}/${port}${path}`;
    return opts;
  },
  headers: {
    'Custom-header': 'x-for-proxy'
  },
  timeout : 1000
});

myInstance.get('http://example.com', function (err, res, data) {})

Timeout

You can set a timeout (in milliseconds) on the request with the timeout option. If the request takes longer than timeout to complete, then the entire request will fail with an Error.

const rock = require('rock-req')

const opts = {
  url: 'http://example.com',
  timeout: 2000 // 2 second timeout
}

rock(opts, function (err, res, data) {})

One Quick Tip

It's a good idea to set the 'user-agent' header so the provider can more easily see how their resource is used.

const rock = require('rock-req')
const pkg = require('./package.json')

rock({
  url : 'http://example.com',
  headers: {
    'user-agent': `my-module/${pkg.version} (https://github.com/username/my-module)`
  }
}, function (err, res, data) {})

Proxies

You can use the tunnel module with the agent option to work with proxies:

const rock = require('rock-req')
const tunnel = require('tunnel')

const opts = {
  url: 'http://example.com',
  agent: tunnel.httpOverHttp({
    proxy: {
      host: 'localhost'
    }
  })
}

rock(opts, function (err, res, data) {})

Cookies

You can use the cookie module to include cookies in a request:

You can extend and create a new instance of rock-req to keep the cookie in header for each request.

const rock = require('rock-req')
const cookie = require('cookie')

const opts = {
  url: 'http://example.com',
  headers: {
    cookie: cookie.serialize('foo', 'bar')
  }
}

rock(opts, function (err, res, data) {})

Form data

You can use the form-data module to create POST request with form data:

const fs = require('fs')
const rock = require('rock-req')
const FormData = require('form-data')
const form = new FormData()

const opts = {
  url: 'http://example.com',
  body: () => {
    form.append('my_file', fs.createReadStream('/foo/bar.jpg'))
  }
}

rock.post(opts, function (err, res, data) {})

Or, include application/x-www-form-urlencoded form data manually:

const rock = require('rock-req')

const opts = {
  url: 'http://example.com',
  form: {
    key: 'value'
  }
}
rock.post(opts, function (err, res, data) {})

Specifically disallowing redirects

const rock = require('rock-req')

const opts = {
  url: 'http://example.com/will-redirect-elsewhere',
  followRedirects: false
}
// res.statusCode will be 301, no error thrown
rock(opts, function (err, res, data) {})

Basic Auth

const user = 'someuser'
const pass = 'pa$$word'
const encodedAuth = Buffer.from(`${user}:${pass}`).toString('base64')

rock('http://example.com', {
  headers: {
    authorization: `Basic ${encodedAuth}`
  }
})

OAuth

You can use the oauth-1.0a module to create a signed OAuth request:

const rock = require('rock-req')
const crypto  = require('crypto')
const OAuth = require('oauth-1.0a')

const oauth = OAuth({
  consumer: {
    key: process.env.CONSUMER_KEY,
    secret: process.env.CONSUMER_SECRET
  },
  signature_method: 'HMAC-SHA1',
  hash_function: (baseString, key) => crypto.createHmac('sha1', key).update(baseString).digest('base64')
})

const token = {
  key: process.env.ACCESS_TOKEN,
  secret: process.env.ACCESS_TOKEN_SECRET
}

const url = 'https://api.twitter.com/1.1/statuses/home_timeline.json'

const opts = {
  url: url,
  headers: oauth.toHeader(oauth.authorize({url, method: 'GET'}, token)),
  json: true
}

rock(opts, function (err, res) {})

Throttle requests

You can use limiter to throttle requests. This is useful when calling an API that is rate limited.

const rock = require('rock-get')
const RateLimiter = require('limiter').RateLimiter
const limiter = new RateLimiter(1, 'second')

const rock = (opts, cb) => limiter.removeTokens(1, () => rock(opts, cb))
rock.concat = (opts, cb) => limiter.removeTokens(1, () => rock.concat(opts, cb))

var opts = {
  url: 'http://example.com'
}

rock.concat(opts, processResult)
rock.concat(opts, processResult)

function processResult (err, res, data) {
  if (err) throw err
  console.log(data.toString())
}

Breaking change for people coming from simplet-get

Rock-req is a fork of simple-get

• Rock-req concatenates chunks and returns the concatenated buffer/JSON in the third paramater of the callback. Why? because 80% of the time, we need to do simple request without streams. When you need to pipe the result to another stream, you must use opts.output parameter (see "Output Stream" in the doc)
• All streams must be created in a function. Rock-req return an error if it is not the case
  • before: body = stream
  • after body = () => { const myStream = create(); return myStream; }

Notes:

• replace deprecated url.parse by new URL but new URL is slower than url.parse. Let's see if Node 20 LTS is faster
• agent keep Alive
• add advanced timeout (response timeout)
• test prevError

Supporters

This packaged in maintained by Carbone:

Thank you Feross Aboukhadijeh, creator of simple-get