Request-libcurl
npm install --save request-libcurl
This is a server-only package. This package was created due to a lack of stability in the Node's http
/https
ClientRequest modules. Since we've been looking for something tested by decades and generations, — our choice stopped on libcurl
, later core library might be changed, but we would keep same API and idea about fast, sustainable and simple HTTP requests.
Main features
- 👨💻 98% tests coverage + TDD (only for http(s));
- 👷♂️ Follow
request
API (simplified); - 📦 The single dependency on
node-libcurl
package; - 😎 IDNs support (internationalized domain names);
- 😎 HTTP/2 support;
- 😎 Repeat (built-in retries) request on failed or broken connection;
- 😎 Send GET/POST with custom
body
and headers; - 😎 Follow or deny redirects;
- 📤 Upload files with a single line;
- 🔐 Ignore or deny "broken" SSL/TLS certificates;
- 💪 Bulletproof design, during development we avoid complex solutions.
ToC:
Install
# ONLY for node@>=8.9.0
npm install request-libcurl --save
const request = require('request-libcurl');
import request from 'request-libcurl';
Note
We build this package to serve our needs and solve our issues with Node's native API. It may have a lack of compatibility with request()
module API, or compatible only partially.
API
const request = require('request-libcurl');
const opts = {
method: 'GET',
url: 'https://example.com',
auth: 'username:password',
form: '{"ops": "value"}',
headers: {
Accept: '*/*',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'
},
debug: false,
retry: true,
retries: 3,
timeout: 6144,
keepAlive: false,
retryDelay: 256,
followRedirect: true,
maxRedirects: 4,
rejectUnauthorized: false
};
request(opts, (error, resp) => {
if (error) {
const { errorCode, code, statusCode, message } = error;
} else {
const { statusCode, body, headers } = resp;
}
});
Request default options
const request = require('request-libcurl');
request.defaultOptions = {
wait: false,
proxy: false,
retry: true,
debug: false,
method: 'GET',
timeout: 6144,
retries: 3,
rawBody: false,
keepAlive: false,
noStorage: false,
retryDelay: 256,
maxRedirects: 4,
followRedirect: true,
rejectUnauthorized: false,
rejectUnauthorizedProxy: false,
badStatuses: [ 300, 303, 305, 400, 407, 408, 409, 410, 500, 502, 503, 504, 510 ],
isBadStatus(statusCode, badStatuses = request.defaultOptions.badStatuses) {
return badStatuses.includes(statusCode) || statusCode >= 500;
},
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
Accept: '*/*'
}
};
request.defaultOptions.timeout = 7000;
request.defaultOptions.retries = 12;
request.defaultOptions.retryDelay = 5000;
request.defaultOptions.followRedirect = false;
request.defaultOptions.badStatuses = [300, 303, 305, 400, 407, 408, 409, 410];
request.defaultOptions.isBadStatus = (statusCode, badStatuses = request.defaultOptions.badStatuses) => {
return badStatuses.includes(statusCode) || statusCode >= 500;
};
Request options
opts.url
or opts.uri
{String} - [Required] Fully qualified URI with protocol http
/https
;opts.method
{String} - [Optional] HTTP Method name, you can use any valid method name from HTTP specs, tested with GET/POST, default: GET
;opts.auth
{String} - [Optional] value for HTTP Authorization header as plain string in a form of username:password
;opts.form
{String|Object} - [Optional] Custom request body for POST request. If {String} is passed Content-Type
will be set to application/x-www-form-urlencoded
, by passing plain {Object} Content-Type
will be set to application/json
. To set custom Content-Type
— pass it to opts.headers
Object;opts.upload
{Integer} - [Optional] To upload a file pass an Integer representing the file descriptor. See this example for reference;opts.headers
{Object} - [Optional] Custom request headers, default: { Accept: '*/*', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36' }
. Note: setting custom request headers will replace default ones;opts.debug
{Boolean} - [Optional] Enable debug and extra logging, default: false
;opts.retry
{Boolean} - [Optional] Retry request if connection is broken? Default: true
;opts.retries
{Number} - [Optional] How many times retry request if connection is broken, default: 3
;opts.retryDelay
{Number} - [Optional] How long to wait between request retries (ms), default: 256
;opts.timeout
{Number} - [Optional] How long to wait for response (ms), default: 6144
;opts.followRedirect
{Boolean} - [Optional] Shall request follow redirects? Default: true
;opts.keepAlive
{Boolean} - [Optional] Turn on TCP keepalive probes, default: false
;opts.maxRedirects
{Number} - [Optional] How many redirects are supported during single request, default: 4
;opts.badStatuses
{[Number]} - [Optional] Array of "bad" status codes responsible for triggering request retries, default: [300, 303, 305, 400, 407, 408, 409, 410, 500, 502, 503, 504, 510]
;opts.isBadStatus
{Function} - [Optional] Function responsible for triggering request retries, default (at the bottom of code-block);opts.rawBody
{Boolean} - Disable all data processing (body
will be passed as Buffer, headers
will be empty, use .onHeader()
hook to get headers with rawBody
option), great option for piping, default: false
;opts.noStorage
{Boolean} - Disable all data processing and data concatenation (headers
and body
won't be passed to response), great option for piping, default: false
;opts.wait
{Boolean} - Do not send request immediately and wait until .send()
method is called, set this option to true
to register .onHeader()
and .onBody()
hooks, default: false
;opts.proxy
{String} - Fully qualified URL to HTTP proxy, when this feature is enabled connections are going to start with CONNECT
request, default: no proxy or system proxy is used;opts.rejectUnauthorized
{Boolean} - [Optional] Shall request be rejected if SSL/TLS certificate can't be validated? Default: false
;opts.rejectUnauthorizedProxy
{Boolean} - [Optional] Shall request be rejected if SSL/TLS certificate of a proxy host can't be validated? Default: false
;opts.curlOptions
{Object} - [Optional] Explicitly set libcurl
options, full list of options available here and here;opts.curlFeatures
{Object} - [Optional] Explicitly enable or disable libcurl
features. To enable a feature pass true
as a value, example: {NoDataParsing: true}
. To disable pass false
as a value, example: {NoDataParsing: false}
. Full list of available features is available here.
Notes:
- When using
opts.rawBody
callback won't return headers
, to get headers use onHeader
hook; - When using
opts.noStorage
callback won't return headers
and body
, to get headers and body use onData
and onHeader
hooks; opts.upload
and opts.form
can not be used together, there won't be exception thrown, if both presented — opts.form
will be used;- When using
opts.upload
or any other request where server returns expect: '100-continue'
HTTP header — callback won't return headers
, to get headers use onHeader
hook; - This package is build on top of
libcurl
and node-libcurl
it's the way much more powerful than just sending requests via http
and https
protocol. Libcurl can work with IMAP/SMTP protocols getting/sending emails. Libcurl can serve as fully-featured FTP-client. Here's full list of supported protocols: DICT
, FILE
, FTP
, FTPS
, Gopher
, HTTP
, HTTPS
, IMAP
, IMAPS
, LDAP
, LDAPS
, POP3
, POP3S
, RTMP
, RTSP
, SCP
, SFTP
, SMTP
, SMTPS
, Telnet
and TFTP
. To learn more on how to utilize all available power and features see docs of node-libcurl
and libcurl
itself.
let _body = Buffer.from('');
let _headers = Buffer.from('');
const headersObj = {};
const req = request({
url: 'https://example.com',
retry: false,
rawBody: true,
wait: true
}, (error) => {
if (error) {
throw error;
}
const body = _body.toString('utf8');
const headers = _headers.toString('utf8');
});
req.onData((chunkAsBuffer) => {
_body = Buffer.concat([_body, chunkAsBuffer]);
});
req.onHeader((chunkAsBuffer) => {
_headers = Buffer.concat([_headers, chunkAsBuffer]);
const header = chunkAsBuffer.toString('utf8');
if (header.includes(':')) {
const splitHeader = header.split(':');
headersObj[splitHeader[0].toLowerCase().trim()] = splitHeader[1].trim();
}
});
req.send();
Response
resp.statusCode
{Number} - HTTP response/status code;resp.body
{String} - Body of HTTP response, not modified or processed, as it is — plain text;resp.headers
{Object} - HTTP response headers as plain Object, all headers names are lower-cased.
Error
error.errorCode
{Number} - libcurl
internal error code;error.code
{Number} - libcurl
internal error code, same as errorCode
;error.statusCode
{Number} - HTTP error code, if any;error.message
{String} - Human-readable error.
Returns req
Object
const request = require('request-libcurl');
const req = request({url: 'https://example.com'});
req.abort()
- Abort current request, request will return 499: Client Closed Request
HTTP errorreq.send()
- Send request, use it with wait
. For example with rawBody
/noStorage
, when you need to delay sending request, for example to set event listeners and/or hooksreq.onData(callback)
- Hook, called right after data is received, called for each data-chunk. Useful with .pipe()
, rawBody
/noStorage
and hooks/eventsreq.onHeader(callback)
- Hook, called right after header is received, called for each header. Useful with .pipe()
, rawBody
/noStorage
and hooks/eventscallback(error, resp)
- Callback triggered on successful response
error
{undefined};resp.statusCode
{Number} - HTTP status code;resp.body
{String} - Body of HTTP response, not modified or processed, as it is — plain text;resp.headers
{Object} - Key-value plain Object with pairs of response headers;
callback(error)
- Callback triggered on failed request
error.errorCode
{Number} - libcurl
internal error code;error.code
{Number} - libcurl
internal error code, same as errorCode
;error.statusCode
{Number} - HTTP error code, if any;error.message
{String} - Human-readable error.
Examples
GET request
const request = require('request-libcurl');
request({ url: 'https://example.com' }, (error, resp) => {
});
POST request
const request = require('request-libcurl');
const querystring = require('querystring');
request({
method: 'POST',
url: 'https://example.com',
form: querystring.stringify({ myForm: 'data' })
}, (error, resp) => {
});
request({
method: 'POST',
url: 'https://example.com',
auth: 'username:passwd',
form: querystring.stringify({ myForm: 'data' })
}, (error, resp) => {
});
request({
method: 'POST',
url: 'https://example.com',
form: { myForm: 'data' }
}, (error, resp) => {
});
const request = require('request-libcurl');
request({
method: 'POST',
url: 'https://example.com',
auth: 'username:passwd',
form: { myForm: 'data' }
}, (error, resp) => {
});
request({
method: 'POST',
url: 'https://example.com',
form: 'Plain String or Base64 String or any other String',
headers: {
'Content-Type': 'text/plain'
}
}, (error, resp) => {
});
File upload
const fs = require('fs');
const request = require('request-libcurl');
fs.open('/path/to/a/file', 'r', function(err, fd) {
if (err) {
throw new Error('can not read the file');
}
request({
method: 'POST',
url: 'https://example.com/upload',
upload: fd,
retry: false,
}, (error, resp) => {
if (error) {
throw error;
} else {
}
});
});
File upload (multipart/form-data
)
In this example we are going to use HTTPPOST
libcurl option passing [Object]
(array of Objects representing files, note: multiple files can be passed in a single request) via curlOptions
const request = require('request-libcurl');
const fileLocation = '/full/absolute/path/to/a/file.ext';
request({
method: 'POST',
url: 'https://example.com/upload.php',
retry: false,
curlOptions: {
HTTPPOST: [{
name: 'file.ext',
file: fileLocation,
type: 'application/ext'
} ]
}
}, (error) => {
if (error) {
throw error;
} else {
}
});
Running Tests
- Clone this package
- In Terminal (Console) go to directory where package is cloned
- Then run:
# Install development NPM dependencies:
npm install --save-dev
# Install NPM dependencies:
npm install --save
# Run tests:
PORT=3003 npm test
# PORT env.var is required! And can be changed to any open port!
# Note: The Internet connection is required to perform tests
# Note: Test-suite includes "no response" and "timeouted responces"
# if a test looks stuck — give it another minute before interrupting it
Support our open source contribution