urllib

urllib

Request HTTP URLs in a complex world — basic and digest authentication, redirections, cookies, timeout and more.

Install

npm install urllib --save

Usage

TypeScript and ESM

import { request } from 'urllib';

const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

CommonJS

const { request } = require('urllib');

const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

Global response event

You should create a urllib instance first.

import { HttpClient } from 'urllib';

const httpclient = new HttpClient();
httpclient.on('response', (info) => {
  error: err,
  ctx: args.ctx,
  req: {
    url: url,
    options: options,
    size: requestSize,
  },
  res: res
});

const { data, res } = await httpclient.request('https://nodejs.org');
console.log('status: %s, body size: %d, headers: %j', res.statusCode, data.length, res.headers);

API Doc

Method: async request(url[, options])

Arguments

• url String | Object - The URL to request, either a String or a Object that return by url.parse.
• options Object - Optional
  • method String - Request method, defaults to GET. Could be GET, POST, DELETE or PUT. Alias 'type'.
  • data Object - Data to be sent. Will be stringify automatically.
  • content String | Buffer - Manually set the content of payload. If set, data will be ignored.
  • stream stream.Readable - Stream to be pipe to the remote. If set, data and content will be ignored.
  • writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and callback will be called with data set null after finished writing.
  • files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with multipart/form-data format, base on formstream. If method not set, will use POST method by default.
  • contentType String - Type of request data. Could be json (Notes: not use application/json here). If it's json, will auto set Content-Type: application/json header.
  • dataType String - Type of response data. Could be text or json. If it's text, the callbacked data would be a String. If it's json, the data of callback would be a parsed JSON Object and will auto set Accept: application/json header. Default callbacked data would be a Buffer.
  • fixJSONCtlChars Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is false.
  • headers Object - Request headers.
  • timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Defaults to exports.TIMEOUT, both are 5s. You can use timeout: 5000 to tell urllib use same timeout on two phase or set them seperately such as timeout: [3000, 5000], which will set connecting timeout to 3s and response 5s.
  • auth String - username:password used in HTTP Basic Authorization.
  • agent http.Agent - HTTP Agent object. Set false if you does not use agent.
  • httpsAgent https.Agent - HTTPS Agent object. Set false if you does not use agent.
  • followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to false.
  • maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
  • formatRedirectUrl Function - Format the redirect url by your self. Default is url.resolve(from, to).
  • beforeRequest Function - Before request hook, you can change every thing here.
  • streaming Boolean - let you get the res object when request connected, default false. alias customResponse
  • gzip Boolean - Accept gzip, br response content and auto decode it, default is false.
  • timing Boolean - Enable timing or not, default is false.

Options: options.data

When making a request:

await request('https://example.com', {
  method: 'GET',
  data: {
    'a': 'hello',
    'b': 'world',
  }
});

For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.

For others like POST, PATCH or PUT request, in defaults, the data will be stringify into application/x-www-form-urlencoded format if content-type header is not set.

If content-type is application/json, the data will be JSON.stringify to JSON data format.

Options: options.content

options.content is useful when you wish to construct the request body by yourself, for example making a content-type: application/json request.

Notes that if you want to send a JSON body, you should stringify it yourself:

await request('https://example.com', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  content: JSON.stringify({
    a: 'hello',
    b: 'world',
  })
});

It would make a HTTP request like:

POST / HTTP/1.1
host: example.com
content-type: application/json

{
  "a": "hello",
  "b": "world"
}

This exmaple can use options.data with application/json content type:

await request('https://example.com', {
  method: 'POST',
  headers: {
    'content-type': 'application/json'
  },
  data: {
    a: 'hello',
    b: 'world',
  }
});

Options: options.files

Upload a file with a hello field.

await request('https://example.com/upload', {
  method: 'POST',
  files: __filename,
  data: {
    hello: 'hello urllib',
  },
});

Upload multi files with a hello field.

await request('https://example.com/upload', {
  method: 'POST',
  files: [
    __filename,
    fs.createReadStream(__filename),
    Buffer.from('mock file content'),
  ],
  data: {
    hello: 'hello urllib with multi files',
  },
});

Custom file field name with uploadfile.

await request('https://example.com/upload', {
  method: 'POST',
  files: {
    uploadfile: __filename,
  },
});

Options: options.stream

Uploads a file with formstream:

import formstream from 'formstream';

const form = formstream();
form.file('file', __filename);
form.field('hello', '你好urllib');

await request('https://example.com/upload', {
  method: 'POST',
  headers: form.headers(),
  stream: form,
});

Response Object

Response is normal object, it contains:

• status or statusCode: response status code.
  • -1 meaning some network error like ENOTFOUND
  • -2 meaning ConnectionTimeoutError
• headers: response http headers, default is {}
• size: response size
• aborted: response was aborted or not
• rt: total request and response time in ms.
• timing: timing object if timing enable.
• remoteAddress: http server ip address
• remotePort: http server ip port
• socketHandledRequests: socket already handled request count
• socketHandledResponses: socket already handled response count

Run test with debug log

NODE_DEBUG=urllib npm test

Contributors

fengmk2	dead-horse	xingrz	popomore	JacksonTian	ibigbug
greenkeeperio-bot	atian25	paambaati	denghongcai	XadillaX	alsotang
leoner	hyj1991	isayme	killagu	cyjake	whxaxes
chadxz	danielwpz	danielsss	Jeff-Tian	jedahan	nick-ng
rishavsharan	willizm	davidkhala	aleafs	Amunu	azure-pipelines[bot]
changzhiwin	yuzhigang33	fishbar	gxcsoccer	mars-coder	rockdai
dickeylth	aladdin-add

This project follows the git-contributor spec, auto updated at Tue Jul 05 2022 16:17:31 GMT+0800.

License

MIT

Similar packages

Other packages similar to urllib

axios

Axios is a promise-based HTTP client for the browser and Node.js. It provides a more modern and flexible API compared to urllib, with support for interceptors, request cancellation, and automatic JSON data transformation.

node-fetch

Node-fetch is a lightweight module that brings the Fetch API to Node.js. It is a minimalistic alternative to urllib, focusing on simplicity and compliance with the Fetch standard.

request

Request is a simplified HTTP client for Node.js with a rich set of features. It is more feature-rich than urllib, offering support for OAuth, cookies, and multipart form data. However, it has been deprecated in favor of more modern alternatives like axios.