request-libcurl
Advanced tools
npm install --save request-libcurl
This is a server-only package. This package was created due to a lack of stability in the Node's http/https ClientRequest modules. Since we've been looking for something tested by decades and generations, — our choice stopped on libcurl, later core library might be changed, but we would keep same API and idea about fast, sustainable and simple HTTP requests.
request API (simplified);node-libcurl package;body and headers;# ONLY for node@>=8.9.0
npm install request-libcurl --save
// CommonJS
const request = require('request-libcurl');
//ES6 Style:
import request from 'request-libcurl';
We build this package to serve our needs and solve our issues with Node's native API. It may have a lack of compatibility with request() module API, or compatible only partially.
const request = require('request-libcurl');
const opts = {
method: 'GET', // POST, GET
url: 'https://example.com', // String
auth: 'username:password', // String
form: '{"ops": "value"}', // String, can be JSON or any other type of payload
headers: { // Object
Accept: '*/*',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'
},
debug: false, // Boolean
retry: true, // Boolean
retries: 3, // Number
timeout: 6144, // Number
keepAlive: false, // Boolean
retryDelay: 256, // Number
followRedirect: true, // Boolean
maxRedirects: 4, // Number
rejectUnauthorized: false // Boolean
};
// Callback API
request(opts, (error, resp) => {
if (error) {
// Houston we got a problem! 😱
const { errorCode, code, statusCode, message } = error;
} else {
// We've got successful response! 🥳
const { statusCode, body, headers } = resp;
}
});
const request = require('request-libcurl');
// Default "defaultOptions" Object:
request.defaultOptions = {
wait: false,
proxy: false,
retry: true,
debug: false,
method: 'GET',
timeout: 6144,
retries: 3,
rawBody: false,
keepAlive: false,
noStorage: false,
retryDelay: 256,
maxRedirects: 4,
followRedirect: true,
rejectUnauthorized: false,
rejectUnauthorizedProxy: false,
badStatuses: [ 300, 303, 305, 400, 407, 408, 409, 410, 500, 502, 503, 504, 510 ],
isBadStatus(statusCode, badStatuses = request.defaultOptions.badStatuses) {
return badStatuses.includes(statusCode) || statusCode >= 500;
},
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
Accept: '*/*'
}
};
// Override default settings:
request.defaultOptions.timeout = 7000;
request.defaultOptions.retries = 12;
request.defaultOptions.retryDelay = 5000;
request.defaultOptions.followRedirect = false;
// Override bad statuses codes (used to trigger retries)
request.defaultOptions.badStatuses = [300, 303, 305, 400, 407, 408, 409, 410];
// Override function used to trigger retries based on status code
request.defaultOptions.isBadStatus = (statusCode, badStatuses = request.defaultOptions.badStatuses) => {
return badStatuses.includes(statusCode) || statusCode >= 500;
};
opts.url or opts.uri {String} - [Required] Fully qualified URI with protocol http/https;opts.method {String} - [Optional] HTTP Method name, you can use any valid method name from HTTP specs, tested with GET/POST, default: GET;opts.auth {String} - [Optional] value for HTTP Authorization header as plain string in a form of username:password;opts.form {String|Object} - [Optional] Custom request body for POST request. If {String} is passed Content-Type will be set to application/x-www-form-urlencoded, by passing plain {Object} Content-Type will be set to application/json. To set custom Content-Type — pass it to opts.headers Object;opts.upload {Integer} - [Optional] To upload a file pass an Integer representing the file descriptor. See this example for reference;opts.headers {Object} - [Optional] Custom request headers, default: { Accept: '*/*', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36' }. Note: setting custom request headers will replace default ones;opts.debug {Boolean} - [Optional] Enable debug and extra logging, default: false;opts.retry {Boolean} - [Optional] Retry request if connection is broken? Default: true;opts.retries {Number} - [Optional] How many times retry request if connection is broken, default: 3;opts.retryDelay {Number} - [Optional] How long to wait between request retries (ms), default: 256;opts.timeout {Number} - [Optional] How long to wait for response (ms), default: 6144;opts.followRedirect {Boolean} - [Optional] Shall request follow redirects? Default: true;opts.keepAlive {Boolean} - [Optional] Turn on TCP keepalive probes, default: false;opts.maxRedirects {Number} - [Optional] How many redirects are supported during single request, default: 4;opts.badStatuses {[Number]} - [Optional] Array of "bad" status codes responsible for triggering request retries, default: [300, 303, 305, 400, 407, 408, 409, 410, 500, 502, 503, 504, 510];opts.isBadStatus {Function} - [Optional] Function responsible for triggering request retries, default (at the bottom of code-block);opts.rawBody {Boolean} - Disable all data processing (body will be passed as Buffer, headers will be empty, use .onHeader() hook to get headers with rawBody option), great option for piping, default: false;opts.noStorage {Boolean} - Disable all data processing and data concatenation (headers and body won't be passed to response), great option for piping, default: false;opts.wait {Boolean} - Do not send request immediately and wait until .send() method is called, set this option to true to register .onHeader() and .onBody() hooks, default: false;opts.proxy {String} - Fully qualified URL to HTTP proxy, when this feature is enabled connections are going to start with CONNECT request, default: no proxy or system proxy is used;opts.rejectUnauthorized {Boolean} - [Optional] Shall request be rejected if SSL/TLS certificate can't be validated? Default: false;opts.rejectUnauthorizedProxy {Boolean} - [Optional] Shall request be rejected if SSL/TLS certificate of a proxy host can't be validated? Default: false;opts.curlOptions {Object} - [Optional] Explicitly set libcurl options, full list of options available here and here;opts.curlFeatures {Object} - [Optional] Explicitly enable or disable libcurl features. To enable a feature pass true as a value, example: {NoDataParsing: true}. To disable pass false as a value, example: {NoDataParsing: false}. Full list of available features is available here.Notes:
opts.rawBody callback won't return headers, to get headers use onHeader hook;opts.noStorage callback won't return headers and body, to get headers and body use onData and onHeader hooks;opts.upload and opts.form can not be used together, there won't be exception thrown, if both presented — opts.form will be used;opts.upload or any other request where server returns expect: '100-continue' HTTP header — callback won't return headers, to get headers use onHeader hook;libcurl and node-libcurl it's the way much more powerful than just sending requests via http and https protocol. Libcurl can work with IMAP/SMTP protocols getting/sending emails. Libcurl can serve as fully-featured FTP-client. Here's full list of supported protocols: DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. To learn more on how to utilize all available power and features see docs of node-libcurl and libcurl itself.let _body = Buffer.from('');
let _headers = Buffer.from('');
const headersObj = {};
const req = request({
url: 'https://example.com',
retry: false, // Do not retry with rawBody/noStorage, as it may mess up with headers and body inside `.onData()` and `.onHeader()` hooks
rawBody: true,
wait: true // Using 'wait' option to set `.onData()` and `.onHeader()` hooks
}, (error) => {
if (error) {
throw error;
}
const body = _body.toString('utf8');
const headers = _headers.toString('utf8');
});
req.onData((chunkAsBuffer) => {
// Do something with a body
// .pipe() for example
_body = Buffer.concat([_body, chunkAsBuffer]);
});
req.onHeader((chunkAsBuffer) => {
_headers = Buffer.concat([_headers, chunkAsBuffer]);
// or convert it to headers Object:
const header = chunkAsBuffer.toString('utf8');
if (header.includes(':')) {
const splitHeader = header.split(':');
headersObj[splitHeader[0].toLowerCase().trim()] = splitHeader[1].trim();
}
});
req.send();
resp.statusCode {Number} - HTTP response/status code;resp.body {String} - Body of HTTP response, not modified or processed, as it is — plain text;resp.headers {Object} - HTTP response headers as plain Object, all headers names are lower-cased.error.errorCode {Number} - libcurl internal error code;error.code {Number} - libcurl internal error code, same as errorCode;error.statusCode {Number} - HTTP error code, if any;error.message {String} - Human-readable error.req Objectconst request = require('request-libcurl');
const req = request({url: 'https://example.com'});
req.abort() - Abort current request, request will return 499: Client Closed Request HTTP errorreq.send() - Send request, use it with wait. For example with rawBody/noStorage, when you need to delay sending request, for example to set event listeners and/or hooksreq.onData(callback) - Hook, called right after data is received, called for each data-chunk. Useful with .pipe(), rawBody/noStorage and hooks/eventsreq.onHeader(callback) - Hook, called right after header is received, called for each header. Useful with .pipe(), rawBody/noStorage and hooks/eventscallback(error, resp) - Callback triggered on successful response
error {undefined};resp.statusCode {Number} - HTTP status code;resp.body {String} - Body of HTTP response, not modified or processed, as it is — plain text;resp.headers {Object} - Key-value plain Object with pairs of response headers;callback(error) - Callback triggered on failed request
error.errorCode {Number} - libcurl internal error code;error.code {Number} - libcurl internal error code, same as errorCode;error.statusCode {Number} - HTTP error code, if any;error.message {String} - Human-readable error.const request = require('request-libcurl');
// GET request:
request({ url: 'https://example.com' }, (error, resp) => {
/* ... */
});
const request = require('request-libcurl');
const querystring = require('querystring');
// POST (Content-Type: application/x-www-form-urlencoded):
// by passing a String or formatted "Query String" to `form`
request({
method: 'POST',
url: 'https://example.com',
form: querystring.stringify({ myForm: 'data' })
}, (error, resp) => {
/* ... */
});
// POST with Authorization (Content-Type: application/x-www-form-urlencoded):
// by passing a String or formatted "Query String" to `form`
request({
method: 'POST',
url: 'https://example.com',
auth: 'username:passwd',
form: querystring.stringify({ myForm: 'data' })
}, (error, resp) => {
/* ... */
});
// POST (Content-Type: application/json):
// by passing plain Object to `form`
request({
method: 'POST',
url: 'https://example.com',
form: { myForm: 'data' }
}, (error, resp) => {
/* ... */
});
const request = require('request-libcurl');
// POST with Authorization (Content-Type: application/json):
// by passing plain Object to `form`
request({
method: 'POST',
url: 'https://example.com',
auth: 'username:passwd',
form: { myForm: 'data' }
}, (error, resp) => {
/* ... */
});
// Custom POST (Content-Type: text/plain):
// by passing custom Headers
request({
method: 'POST',
url: 'https://example.com',
form: 'Plain String or Base64 String or any other String',
headers: {
'Content-Type': 'text/plain'
}
}, (error, resp) => {
/* ... */
});
const fs = require('fs');
const request = require('request-libcurl');
fs.open('/path/to/a/file', 'r', function(err, fd) {
if (err) {
throw new Error('can not read the file');
}
request({
method: 'POST',
url: 'https://example.com/upload',
upload: fd,
retry: false,
}, (error, resp) => {
if (error) {
throw error;
} else {
// File successfully uploaded
}
});
});
multipart/form-data)In this example we are going to use HTTPPOST libcurl option passing [Object] (array of Objects representing files, note: multiple files can be passed in a single request) via curlOptions
const request = require('request-libcurl');
const fileLocation = '/full/absolute/path/to/a/file.ext';
request({
method: 'POST', // Can be used with PUT
url: 'https://example.com/upload.php',
retry: false,
curlOptions: {
HTTPPOST: [{
name: 'file.ext', // File's name
file: fileLocation, // Full absolute path to a file on FS
type: 'application/ext' // File's mime-type
} /*, {...} */]
}
}, (error) => {
if (error) {
throw error;
} else {
// File(s) successfully uploaded
}
});
To address most common issue with SSL certificates and speed up response time — SSL/TLS certificates validation is disabled in this package by default. But on edge cases this may return error-code 35 on SNI-enabled hosts. To solve this issue add { rejectUnauthorized: true } to request object.
To change rejectUnauthorized option globally use:
request.defaultOptions.rejectUnauthorized = true;
Due to single dependency on node-libcurl which shipped with statically built binaries, you may encounter This module was compiled against a different Node.js version using NODE_MODULE_VERSION error. This may happen on edge cases, like running the very latest release of node.js (while bundled builds aren't shipped yet), then you may want to build this package locally, use one of next commands:
# Please see options below, in dependence from version of NPM and Node.js
# one of this options should solve this issue
# Option 1: Update and rebuild locally installed binaries
npm rebuild --update-binary --build-from-source
# Option 2: Build library
npm install --save request-libcurl --build-from-source
# Option 3: Build library and curl executables:
npm install --save request-libcurl --build-from-source --curl_static_build=true
# In case if you encounter errors during building package locally:
# 1. Execute same command as "sudo" (e.g. administrator), and try again
# 2. Install globally node-gyp and node-pre-gyp NPM packages, and try again
For more details and instructions for different platforms read node-libcurl official docs. Note: It's highly recommended to run tests after building package locally.
# Install development NPM dependencies:
npm install --save-dev
# Install NPM dependencies:
npm install --save
# Run tests:
PORT=3003 npm test
# Run tests and output debugging details:
DEBUG=true PORT=3003 npm test
# PORT env.var is required! And can be changed to any open port!
# Note: The Internet connection is required to perform tests
# Note: Test-suite includes "no response" and "timing out responses"
# if a test looks stuck — give it another minute before interrupting it
FAQs
Extremely stable HTTP request module built on top of libcurl
We found that request-libcurl demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Node.js will be enforcing stricter semver-major PR policies a month before major releases to enhance stability and ensure reliable release candidates.
Security News
Research
Socket's threat research team has detected five malicious npm packages targeting Roblox developers, deploying malware to steal credentials and personal data.
Security News
vlt introduced its new package manager and a serverless registry this week, innovating in a space where npm has stagnated.