bhttp

bhttp

A sane HTTP client library for Node.js with Streams2 support.

Why bhttp?

There are already a few commonly used HTTP client libraries for Node.js, but all of them have issues:

• The core http module is rather low-level, and even relatively simple requests take a lot of work to make correctly. It also automatically uses a limited amount of agents for HTTP requests (in Node.js 0.10), which slows down concurrent HTTP requests when you're streaming the responses somewhere.
• request is buggy, only supports old-style streams, has the same 'agent' problem as http, the documentation is poor, and the API is not very intuitive.
• needle is a lot simpler, but suffers from the same 'agent' problem, and the API can be a bit annoying in some ways. It also doesn't have a proper session API.
• hyperquest (mostly) solves the 'agent' problem correctly, but has a very spartan API. Making non-GET requests is more complex than it should be.

All these issues (and more) are solved in bhttp. It offers the following:

• A simple, well-documented API.
• Sane default behaviour.
• Minimal behind-the-scenes 'magic', meaning less opportunities for bugs to be introduced. No 'gotchas' in dealing with response streams either.
• Support for multipart/form-data (eg. file uploads), with support for Streams2, and support for legacy streams.
• Fully automatic detection of desired payload type - URL-encoded, multipart/form-data, or even a stream or Buffer directly. Just give it the data you want to send, and it will make sure it arrives correctly. Optionally, you can also specify JSON encoding (for JSON APIs).
• Easy-to-use session mechanics - a new session will automatically give you a new cookie jar, cookies are kept track of automatically, and 'default options' are deep-merged.
• Streaming requests are kept out of the agent pool - ie. no blocking of other requests.
• Optionally, a Promises API (you can also use nodebacks).
• Progress events! For both uploading and downloading.

Caveats

bhttp does not yet use a HTTPS-capable agent. This means that all SSL-related options are currently ignored by default (per Node.js http documentation).

This does not mean that you cannot use bhttp for HTTPS requests! If you need secure HTTPS requests, just make sure to specify a custom https agent.

License

WTFPL or CC0, whichever you prefer. A donation and/or attribution are appreciated, but not required.

Donate

My income consists entirely of donations for my projects. If this module is useful to you, consider making a donation!

You can donate using Bitcoin, PayPal, Gratipay, Flattr, cash-in-mail, SEPA transfers, and pretty much anything else.

Contributing

Pull requests welcome. Please make sure your modifications are in line with the overall code style, and ensure that you're editing the src/ files, not the lib/ files.

Build tool of choice is babel; simply run npm run build while developing, and it will watch for changes.

Be aware that by making a pull request, you agree to release your modifications under the licenses stated above.

Usage

A simple example:

var Promise = require("bluebird");
var bhttp = require("bhttp");

Promise.try(function() {
	return bhttp.get("http://icanhazip.com/");
}).then(function(response) {
	console.log("Your IP is:", response.body.toString());
});

... or, using nodebacks:

var bhttp = require("bhttp");

bhttp.get("http://icanhazip.com/", {}, function(err, response) {
	console.log("Your IP is:", response.body.toString());
});

Streaming

Demonstrating both streaming responses and using a stream in form data for a request:

var Promise = require("bluebird");
var bhttp = require("bhttp");

Promise.try(function() {
	return bhttp.get("http://somesite.com/bigfile.mp4", {stream: true});
}).then(function(response) {
	return bhttp.post("http://somehostingservice.com/upload", {
		fileOne: response,
		fileTwo: fs.createReadStream("./otherbigfile.mkv")
	});
}).then(function(response) {
	console.log("Response from hosting service:", response.body.toString());
});

... or, using nodebacks:

var bhttp = require("bhttp");

bhttp.get("http://somesite.com/bigfile.mp4", {stream: true}, function(err, responseOne) {
	var payload = {
		fileOne: responseOne,
		fileTwo: fs.createReadStream("./otherbigfile.mkv")
	};

	bhttp.post("http://somehostingservice.com/upload", payload, {}, function(err, responseTwo) {
		console.log("Response from hosting service:", responseTwo.body.toString());
	})
})

Progress events

Upload progress events:

var Promise = require("bluebird");
var bhttp = require("bhttp");

Promise.try(function() {
	return bhttp.post("http://somehostingservice.com/upload", {
		file: fs.createReadStream("./bigfile.mkv")
	}, {
		onUploadProgress: function(completedBytes, totalBytes) {
			console.log("Upload progress:", (completedBytes / totalBytes * 100), "%");
		}
	});
}).then(function(response) {
	console.log("Response from hosting service:", response.body.toString());
});

Download progress events:

var Promise = require("bluebird");
var bhttp = require("bhttp");

Promise.try(function() {
	return bhttp.get("http://somehostingservice.com/bigfile.mkv", {stream: true});
}).then(function(response) {
	response.on("progress", function(completedBytes, totalBytes) {
		console.log("Download progress:", (completedBytes / totalBytes * 100), "%");
	});

	response.pipe(fs.createWriteStream("./bigfile.mkv"));
});

Sessions

var Promise = require("bluebird");
var bhttp = require("bhttp");

var session = bhttp.session({ headers: {"user-agent": "MyCustomUserAgent/2.0"} });

// Our new session now automatically has a cookie jar, and also uses our preset option(s).

Promise.try(function(){
	return session.get("http://hypotheticalsite.com/cookietest"); // Assume that this site now sets a cookie
}).then(function(response){
	return session.get("http://hypotheticalsite.com/other-endpoint"); // This now sends along the cookie!
});

API

The various error types are documented at the bottom of this README.

bhttp.head(url, [options, [callback]])

bhttp.get(url, [options, [callback]])

bhttp.delete(url, [options, [callback]])

bhttp.post(url, [data, [options, [callback]]])

bhttp.put(url, [data, [options, [callback]]])

bhttp.patch(url, [data, [options, [callback]]])

Convenience methods that pre-set the request method, and automatically send along the payload using the correct options for bhttp.request.

• url: The URL to request, with protocol. When using HTTPS, please be sure to read the 'Caveats' section.
• data: Optional, only for POST/PUT/PATCH. The payload to send along.
• options: Optional. Extra options for the request. More details under the documentation for the bhttp.request method below.
• callback: Optional. When using the nodeback API, the callback to use. If not specified, a Promise will be returned.

The data payload can be one of the following things:

• String / Buffer: The contents will be written to the request as-is.
• A stream: The entire stream will be written to the request as-is.
• An object: Will be encoded as form data, and can contain any combination of Strings, Buffers, streams, and arrays of any of those. When only strings are used, the form data is querystring-encoded - if Buffers or streams are used, it will be encoded as multipart/form-data.

Further documentation for these methods, such as the response attributes, can be found in the below section for bhttp.request.

bhttp.request(url, [options, [callback]])

Makes a request, and returns the response object asynchronously. The response object is a standard http.IncomingMessages with a few additional properties (documented below the argument list).

Note that (progress) event handlers must be specified in the options or (in the case of download progress events) as an event listener on the response object - as bhttp uses Promises, it is not technically possible to return an EventEmitter.

• url: The URL to request, with protocol. When using HTTPS, please be sure to read the 'Caveats' section.

• options: Optional. Extra options for the request. Any other options not listed here will be passed on directly to the http or https module.

  • Basic options
    • stream: Defaults to false. Whether the response is meant to be streamed. If true, the response body won't be parsed, an unread response stream is returned, and the request is kept out of the 'agent' pool.
    • headers: Any extra request headers to set. (Non-custom) header names must be lowercase.
    • followRedirects: Defaults to true. Whether to automatically follow redirects or not (the redirect history is available as the redirectHistory property on the response).
    • redirectLimit: Defaults to 10. The maximum amount of redirects to follow before erroring out, to prevent redirect loops.
  • Encoding and decoding
    • forceMultipart: Defaults to false. Ensures that mulipart/form-data encoding is used, no matter what the payload contents are.
    • encodeJSON: Defaults to false. When set to true, the request payload will be encoded as JSON. This cannot be used if you are using any streams in your payload.
    • decodeJSON: Defaults to false. When set to true, the response will always be decoded as JSON, no matter what the Content-Type says. You'll probably want to keep this set to false - most APIs send the correct Content-Type headers, and in those cases bhttp will automatically decode the response as JSON.
    • noDecode: Defaults to false. Never decode the response, even if the Content-Type says that it's JSON.
  • Request payloads (you won't need these when using the shorthand methods)
    • inputBuffer: A Buffer or String to send as the entire payload.
    • inputStream: A stream to send as the entire payload.
    • formFields: Form data to encode. This can also include files to upload.
    • files: Form data to send explicitly as a file. This will automatically enable multipart/form-data encoding.
  • Advanced options
    • method: The request method to use. You don't need this when using the shorthand methods.
    • cookieJar: A custom cookie jar to use. You'll probably want to use bhttp.session() instead.
    • responseTimeout: The timeout, in milliseconds, after which the request should be considered to have failed if no response is received yet. Note that this measures from the start of the request to the start of the response, and is not a connection timeout. If a timeout occurs, a ResponseTimeoutError will be thrown asynchronously (see error documentation below).
    • allowChunkedMultipart: Defaults to false. Many servers don't support multipart/form-data when it is transmitted with chunked transfer encoding (eg. when the stream length is unknown), and silently fail with an empty request payload - this is why bhttp disallows it by default. If you are absolutely certain that the endpoint supports this functionality, you can override the behaviour by setting this to true.
    • discardResponse: Defaults to false. Whether to throw away the response without reading it. Only really useful for fire-and-forget calls. This is almost never what you want.
    • keepRedirectResponses: Defaults to false. Whether to keep the response streams of redirects. You probably don't need this. When enabling this, you must explicitly read out every single redirect response, or you will experience memory leaks!
    • justPrepare: Defaults to false. When set to true, bhttp just prepares the request, and doesn't actually carry it out; useful if you want to make some manual modifications. Instead of a response, the method will asynchronously return an array with the signature [request, response, requestState] that you will need to pass into the bhttp.makeRequest() method.
  • Event handlers
    • onUploadProgress: A callback to call for upload progress events (this covers both input streams and form data). The callback signature is (completedBytes, totalBytes, request). If the total size is not known, totalBytes will be undefined. The request variable will hold the request object that the progress event applies to - this is relevant when dealing with automatic redirect following, where multiple requests may occur.
    • onDownloadProgress: A callback to call for download progress events. The callback signature is (completedBytes, totalBytes, response). If the total size is not known, totalBytes will be undefined. The response variable will hold the response object that the progress event applies to - this is relevant when dealing with automatic redirect following, where multiple responses may occur. Note that using the progress event on a response object is usually a more practical option!

• callback: Optional. When using the nodeback API, the callback to use. If not specified, a Promise will be returned.

A few extra properties are set on the response object (which is a http.IncomingMessage):

• body: When stream is set to false (the default), this will contain the response body. This can be either a Buffer or, in the case of a JSON response, a decoded JSON object.
• redirectHistory: An array containing the redirect responses, if any, in chronological order. Response bodies are discarded by default; if you do not want this, use the keepRedirectResponses option.
• request: The request configuration that was generated by bhttp. You probably don't need this.
• requestState: The request state that was accumulated by bhttp. You probably don't need this.

Additionally, there's an extra event on the response object:

• 'progress' (completedBytes, totalBytes): The 'download progress' for the response body. This works the same as the onDownloadProgress option, except the event will be specific to this response, and it allows for somewhat nicer syntax. Make sure to attach this handler before you start reading the response stream!

bhttp can automatically parse the metadata for the following types of streams:

• fs streams
• http and bhttp responses
• request requests
• combined-stream streams (assuming all the underlying streams are of one of the types listed here)

If you are using a different type of stream, you can wrap the stream using bhttp.wrapStream to manually specify the needed metadata.

bhttp.session([defaultOptions])

This will create a new session. The defaultOptions will be deep-merged with the options specified for each request (where the request-specific options have priority).

A new cookie jar is automatically created, unless you either specify a custom cookieJar option or set the cookieJar option to false (in which case no cookie jar is used).

bhttp.wrapStream(stream, options)

This will return a 'stream wrapper' containing explicit metadata for a stream. You'll need to use it when passing an unsupported type of stream to a data parameter or formFields/files option.

• stream: The stream to wrap.
• options: The options for this stream. All options are optional, but recommended to specify.
  • contentLength: The length of the stream in bytes.
  • contentType: The MIME type of the stream.
  • filename: The filename of the stream.

The resulting wrapper can be passed on to the bhttp methods as if it were a regular stream.

bhttp.makeRequest(request, response, requestState)

When using the justPrepare option, you can use this method to proceed with the request after manual modifications. The function signature is identical to the signature of the array returned when using justPrepare. response will usually be null, but must be passed on as is, to account for future API changes.

Error types

All these correctly extend the Error class - this means that you can use them as a .catch predicate when using Promises, and that you can use instanceof on them when using the nodeback API.

bhttp.bhttpError

The base class for all errors generated by bhttp. You usually don't need this.

bhttp.ConflictingOptionsError

You have specified two or more request options that cannot be used together.

The error message will contain more details.

bhttp.UnsupportedProtocolError

You tried to load a URL that isn't using either the HTTP or HTTPS protocol. Only HTTP and HTTPS are currently supported.

bhttp.RedirectError

A redirect was encountered that could not be followed.

This could be because the redirect limit was reached, or because the HTTP specification doesn't allow automatic following of the redirect that was encountered.

The error message will contain more details.

bhttp.MultipartError

Something went wrong while generating the multipart/form-data stream.

Currently, this will only be thrown if you try to use chunked transfer encoding for a multipart stream - a common situation where this can occur, is when you pass in streams with an unknown length.

To resolve this error, you must either explicitly specify the length of the streams using bhttp.wrapStream or, if the target server supports it, enable the allowChunkedMultipart option.

bhttp.ConnectionTimeoutError

The connection timed out.

The connection timeout is defined by the operating system, and cannot currently be overridden.

bhttp.ResponseTimeoutError

The response timed out.

The response timeout can be specified using the responseTimeout option, and it is measured from the start of the request to the start of the response. If no response is received within the responseTimeout, a ResponseTimeoutError will be thrown asynchronously, and the request will be aborted.

You should not set a responseTimeout for requests that involve large file uploads! Because a response can only be received after the request has completed, any file/stream upload that takes longer than the responseTimeout, will result in a ResponseTimeoutError.