urllib

What is urllib?

The urllib npm package is a utility library for making HTTP requests and handling URLs. It provides a simple and flexible API for performing various types of HTTP requests, handling query parameters, and working with URLs.

What are urllib's main functionalities?

HTTP GET Request

This feature allows you to make a simple HTTP GET request to a specified URL. The callback function handles the response data or any errors that occur.

const urllib = require('urllib');

urllib.request('https://api.example.com/data', function (err, data, res) {
  if (err) {
    console.error(err);
    return;
  }
  console.log(data.toString());
});

HTTP POST Request

This feature allows you to make an HTTP POST request with a payload. The data object contains the key-value pairs to be sent in the request body.

const urllib = require('urllib');

urllib.request('https://api.example.com/data', {
  method: 'POST',
  data: {
    key1: 'value1',
    key2: 'value2'
  }
}, function (err, data, res) {
  if (err) {
    console.error(err);
    return;
  }
  console.log(data.toString());
});

Handling Query Parameters

This feature allows you to include query parameters in your HTTP request. The data object is automatically serialized into a query string and appended to the URL.

const urllib = require('urllib');

const params = {
  key1: 'value1',
  key2: 'value2'
};

urllib.request('https://api.example.com/data', {
  data: params
}, function (err, data, res) {
  if (err) {
    console.error(err);
    return;
  }
  console.log(data.toString());
});

Custom Headers

This feature allows you to set custom headers for your HTTP request. The headers object contains key-value pairs representing the header names and values.

const urllib = require('urllib');

urllib.request('https://api.example.com/data', {
  headers: {
    'Authorization': 'Bearer token',
    'Content-Type': 'application/json'
  }
}, function (err, data, res) {
  if (err) {
    console.error(err);
    return;
  }
  console.log(data.toString());
});

Other packages similar to urllib

axios

Axios is a promise-based HTTP client for the browser and Node.js. It provides a more modern and flexible API compared to urllib, with support for interceptors, request cancellation, and automatic JSON data transformation.

node-fetch

Node-fetch is a lightweight module that brings the Fetch API to Node.js. It is a minimalistic alternative to urllib, focusing on simplicity and compliance with the Fetch standard.

request

Request is a simplified HTTP client for Node.js with a rich set of features. It is more feature-rich than urllib, offering support for OAuth, cookies, and multipart form data. However, it has been deprecated in favor of more modern alternatives like axios.

README

urllib

Help in opening URLs (mostly HTTP) in a complex world — basic and digest authentication, redirections, cookies, timeout and more.

Install

$ npm install urllib --save

Usage

callback

var urllib = require('urllib');

urllib.request('http://cnodejs.org/', { wd: 'nodejs' }, function (err, data, res) {
  if (err) {
    throw err; // you need to handle error
  }
  console.log(res.statusCode);
  console.log(res.headers);
  // data is Buffer instance
  console.log(data.toString());
});

Promise

If you've installed bluebird, bluebird will be used. urllib does not install bluebird for you.

Otherwise, if you're using a node that has native v8 Promises (v0.11.13+), then that will be used.

Otherwise, this library will crash the process and exit, so you might as well install bluebird as a dependency!

var urllib = require('urllib');

urllib.request('http://nodejs.org').then(function (result) {
  // result: {data: buffer, res: response object}
  console.log('status: %s, body size: %d, headers: %j', result.res.statusCode, result.data.length, result.res.headers);
}).error(function (err) {
  console.error(err);
});

co & generator

If you are using co or koa:

var co = require('co');
var urllib = require('urllib');

co(function* () {
  var result = yield urllib.request('http://nodejs.org');
  console.log('status: %s, body size: %d, headers: %j',
    result.status, result.data.length, result.headers);
})();

Global response event

urllib.on('response', function (info) {
  error: err,
  req: {
    url: url,
    options: options,
    size: requestSize,
  },
  res: res
});

API Doc

Method: http.request(url[, options][, callback])

Arguments

• url String | Object - The URL to request, either a String or a Object that return by url.parse.
• options Object - Optional
  • method String - Request method, defaults to GET. Could be GET, POST, DELETE or PUT. Alias 'type'.
  • data Object - Data to be sent. Will be stringify automatically.
  • content String | Buffer - Manually set the content of payload. If set, data will be ignored.
  • stream stream.Readable - Stream to be pipe to the remote. If set, data and content will be ignored.
  • writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and callback will be called with data set null after finished writing.
  • dataType String - Type of response data. Could be text or json. If it's text, the callbacked data would be a Buffer. If it's json, the data of callback would be a parsed JSON Object. Defaults to text.
  • headers Object - Request headers.
  • timeout Number - Request timeout in milliseconds. Defaults to exports.TIMEOUT. Include remote server connecting timeout and response timeout. When timeout happen, will return ConnectionTimeout or ResponseTimeout.
  • auth String - username:password used in HTTP Basic Authorization.
  • digestAuth String - username:password used in HTTP Digest Authorization.
  • agent http.Agent - HTTP Agent object. Set false if you does not use agent.
  • httpsAgent https.Agent - HTTPS Agent object. Set false if you does not use agent.
  • ca String | Buffer | Array - An array of strings or Buffers of trusted certificates. If this is omitted several well known "root" CAs will be used, like VeriSign. These are used to authorize connections. Notes: This is necessary only if the server uses the self-signed certificate
  • rejectUnauthorized Boolean - If true, the server certificate is verified against the list of supplied CAs. An 'error' event is emitted if verification fails. Default: true.
  • pfx String | Buffer - A string or Buffer containing the private key, certificate and CA certs of the server in PFX or PKCS12 format.
  • key String | Buffer - A string or Buffer containing the private key of the client in PEM format. Notes: This is necessary only if using the client certificate authentication
  • cert String | Buffer - A string or Buffer containing the certificate key of the client in PEM format. Notes: This is necessary only if using the client certificate authentication
  • passphrase String - A string of passphrase for the private key or pfx.
  • followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to false.
  • maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
  • beforeRequest Function - Before request hook, you can change every thing here.
  • gzip Boolean - Accept gzip response content and auto decode it, default is false.
• callback(err, data, res) Function - Optional callback.
  • err Error - Would be null if no error accured.
  • data Buffer | Object - The data responsed. Would be a Buffer if dataType is set to text or an JSON parsed into Object if it's set to json.
  • res http.IncomingMessage - The response.

Returns

http.ClientRequest - The request.

Calling .abort() method of the request stream can cancel the request.

Options: options.data

When making a request:

urllib.request('http://example.com', {
  method: 'GET',
  data: {
    'a': 'hello',
    'b': 'world'
  }
});

For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.

For others like POST, PATCH or PUT request, in defaults, the data will be stringify into application/x-www-form-urlencoded format if Content-Type header is not set.

If Content-type is application/json, the data will be JSON.stringify to JSON data format.

Options: options.content

options.content is useful when you wish to construct the request body by yourself, for example making a Content-Type: application/json request.

Notes that if you want to send a JSON body, you should stringify it yourself:

urllib.request('http://example.com', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  content: JSON.stringify({
    a: 'hello',
    b: 'world'
  })
});

It would make a HTTP request like:

POST / HTTP/1.1
Host: example.com
Content-Type: application/json

{
  "a": "hello",
  "b": "world"
}

This exmaple can use options.data with application/json content type:

urllib.request('http://example.com', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  data: {
    a: 'hello',
    b: 'world'
  }
});

Options: options.stream

Uploads a file with formstream:

var urllib = require('urllib');
var formstream = require('formstream');

var form = formstream();
form.file('file', __filename);
form.field('hello', '你好urllib');

var req = urllib.request('http://my.server.com/upload', {
  method: 'POST',
  headers: form.headers(),
  stream: form
}, function (err, data, res) {
  // upload finished
});

Response Object

Response is normal object, it contains:

• status or statusCode: response status code, -1 meaning some network error like ENOTFOUND
• headers: response http headers, default is {}
• size: response size
• aborted: response was aborted or not
• rt: total request and response time in ms.

Response: res.aborted

If the underlaying connection was terminated before response.end() was called, res.aborted should be true, and return RemoteSocketClosedError error.

var server = require('http').createServer(function (req, res) {
  req.resume();
  req.on('end', function () {
    res.write('foo haha\n');
    setTimeout(function () {
      res.write('foo haha 2');
      setTimeout(function () {
        res.socket.end();
      }, 300);
    }, 200);
    return;
  });
});

urllib.request('http://127.0.0.1:1984/socket.end', function (err, data, res) {
  err.name.should.equal('RemoteSocketClosedError');
  err.message.should.equal('Remote socket was terminated before `response.end()` was called');
  data.toString().should.equal('foo haha\nfoo haha 2');
  should.ok(res.aborted);
  done();
});

TODO

• Support component
• Browser env use Ajax
• [√] Upload file like form upload
• [√] Auto redirect handle
• [√] https & self-signed certificate
• [√] Connection timeout & Response timeout
• [√] Support Accept-Encoding=gzip by options.gzip = true
• [√] Support Digest access authentication

Authors

Below is the output from git-summary.

$ git summary

 project  : urllib
 repo age : 3 years, 3 months
 active   : 61 days
 commits  : 171
 files    : 22
 authors  :
   144	fengmk2                 84.2%
     9	XiNGRZ                  5.3%
     6	ibigbug                 3.5%
     4	Jackson Tian            2.3%
     3	dead_horse              1.8%
     2	alsotang                1.2%
     1	coderhaoxin             0.6%
     1	aleafs                  0.6%
     1	Jonathan Dahan          0.6%

License

(The MIT License)

Copyright (c) 2011-2014 fengmk2 <[e-mail]>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.