hyperquest

What is hyperquest?

The hyperquest npm package is a streaming HTTP client for Node.js. It is designed to be a lightweight and efficient way to make HTTP requests, with a focus on streaming data directly to and from the network.

What are hyperquest's main functionalities?

Making a GET request

This feature allows you to make a simple GET request to a URL and pipe the response directly to a file. In this example, the response from 'http://www.google.com' is written to 'google.html'.

const hyperquest = require('hyperquest');
const fs = require('fs');

hyperquest('http://www.google.com').pipe(fs.createWriteStream('google.html'));

Making a POST request

This feature allows you to make a POST request and stream data from a file to the server. In this example, the contents of 'file.txt' are sent to 'http://example.com/upload' and the server's response is printed to the console.

const hyperquest = require('hyperquest');
const fs = require('fs');

const req = hyperquest.post('http://example.com/upload');
fs.createReadStream('file.txt').pipe(req);
req.pipe(process.stdout);

Handling errors

This feature demonstrates how to handle errors when making a request. In this example, an error event listener is added to handle any errors that occur during the request to 'http://invalid.url'.

const hyperquest = require('hyperquest');

const req = hyperquest('http://invalid.url');
req.on('error', (err) => {
  console.error('Request failed:', err);
});

Other packages similar to hyperquest

request

The 'request' package is a simplified HTTP client for Node.js, offering a more feature-rich API compared to hyperquest. It supports various HTTP methods, custom headers, cookies, and more. However, it is heavier and less focused on streaming.

axios

The 'axios' package is a promise-based HTTP client for Node.js and the browser. It provides a more modern and flexible API, with built-in support for promises and async/await. It is more feature-rich but also heavier than hyperquest.

node-fetch

The 'node-fetch' package is a lightweight module that brings the Fetch API to Node.js. It is similar to hyperquest in terms of being lightweight and efficient, but it uses the Fetch API syntax, which may be more familiar to developers coming from a browser environment.

README

hyperquest

treat http requests as a streaming transport

The hyperquest api is a subset of request.

This module works in the browser with browserify.

rant

This module disables a lot of infuriating things about core http that WILL cause bugs in your application if you think of http as just another kind of stream:

• http requests have a default idle timeout of 2 minutes. This is terrible if you just want to pipe together a bunch of persistent backend processes over http.

• There is a default connection pool of 5 requests. If you have 5 or more extant http requests, any additional requests will HANG for NO GOOD REASON.

hyperquest turns these annoyances off so you can just pretend that core http is just a fancier version of tcp and not the horrible monstrosity that it actually is.

I have it on good authority that these annoyances will be fixed in node 0.12.

example

simple streaming GET

var hyperquest = require('hyperquest');
hyperquest('http://localhost:8000').pipe(process.stdout);

$ node example/req.js
beep boop

pooling is evil

Now to drive the point home about pooling being evil and almost always never what you want ever.

request has its own forever agent thing that works pretty much the same as node core http.request: the wrong, horrible, broken way.

For instance, the following request code takes 12+ seconds to finish:

var http = require('http');
var request = require('request');

var server = http.createServer(function (req, res) {
    res.write(req.url.slice(1) + '\n');
    setTimeout(res.end.bind(res), 3000);
});

server.listen(5000, function () {
    var pending = 20;
    for (var i = 0; i < 20; i++) {
        var r = request('http://localhost:5000/' + i);
        r.pipe(process.stdout, { end: false });
        r.on('end', function () {
            if (--pending === 0) server.close();
        });
    }
});

process.stdout.setMaxListeners(0); // turn off annoying warnings

substack : example $ time node many_request.js 
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

real    0m12.423s
user    0m0.424s
sys 0m0.048s

Surprising? YES. This is pretty much never what you want, particularly if you have a lot of streaming http API endpoints. Your code will just HANG once the connection pool fills up and it won't start working again until some connections die for whatever reason. I have encountered this so many times in production instances and it is SO hard to track down reliably.

Compare to using hyperquest, which is exactly the same code but it takes 3 seconds instead of 12 to finish because it's not completely self-crippled like request and core http.request.

var http = require('http');
var hyperquest = require('hyperquest');

var server = http.createServer(function (req, res) {
    res.write(req.url.slice(1) + '\n');
    setTimeout(res.end.bind(res), 3000);
});

server.listen(5000, function () {
    var pending = 20;
    for (var i = 0; i < 20; i++) {
        var r = hyperquest('http://localhost:5000/' + i);
        r.pipe(process.stdout, { end: false });
        r.on('end', function () {
            if (--pending === 0) server.close();
        });
    }
});

process.stdout.setMaxListeners(0); // turn off annoying warnings

$ time node many_hyperquest.js 
0
1
2
3
4
5
6
8
9
7
10
11
12
13
14
15
16
17
18
19

real    0m3.284s
user    0m0.288s
sys 0m0.060s

So the other thing is, the justification I've heard supporting this horrible limit-of-5 pooling behavior is "performance". The first example which has been tuned for "performance" takes 12 seconds. The second example that removes these "performance" enhancements takes 3. Some performance improvement INDEED!

methods

var hyperquest = require('hyperquest');

var req = hyperquest(uri, opts={}, cb)

Create an outgoing http request to uri or opts.uri. You need not pass any arguments here since there are setter methods documented below.

Return a readable or duplex stream depending on the opts.method.

Default option values:

• opts.method - "GET"
• opts.headers - {}
• opts.auth - undefined, but is set automatically when the uri has an auth string in it such as "http://user:passwd@host". opts.auth is of the form "user:pass", just like http.request().
• opts.agent - false
• opts.timeout - Math.pow(2, 32) * 1000 The value is passed as an argument to the underlying req.setTimeout() function.
• opts.localAddress - local interface to bind for network connections (Node.js only)

In https mode, you can specify options to the underlying tls.connect() call:

• opts.pfx
• opts.key
• opts.cert
• opts.ca
• opts.ciphers
• opts.rejectUnauthorized
• opts.secureProtocol

The request does not go through until the nextTick so you can set values outside of the opts so long as they are called on the same tick.

Optionally you can pass a cb(err, res) to set up listeners for 'error' and 'response' events in one place.

Note that the optional cb is NOT like request in that hyperquest will not buffer content for you or decode to json or any such magical thing.

req.setHeader(key, value);

Set an outgoing header key to value.

req.setLocation(uri);

Set the location if you didn't specify it in the hyperquest() call.

var req = hyperquest.get(uri, opts, cb)

Return a readable stream from hyperquest(..., { method: 'GET' }).

var req = hyperquest.put(uri, opts, cb)

Return a duplex stream from hyperquest(..., { method: 'PUT' }).

var req = hyperquest.post(uri, opts, cb)

Return a duplex stream from hyperquest(..., { method: 'POST' }).

var req = hyperquest.delete(uri, opts, cb)

Return a readable stream from hyperquest(..., { method: 'DELETE' }).

events

req.on('request', function (req) {})

The 'request' event is fired with the ClientRequest object created as a result of the underlying http.request() call.

req.on('response', function (res) {})

The 'response' event is forwarded from the underlying http.request().

req.on('error', function (res) {})

The 'error' event is forwarded from the underlying http.request().

install

With npm do:

npm install hyperquest

license

MIT