queue

What is queue?

The 'queue' npm package is a fast, robust, and extensible queue implementation for managing a list of tasks in a sequential manner. It allows for asynchronous task processing, concurrency control, timeout for tasks, and pausing/resuming the queue. This package is particularly useful for rate-limiting tasks or operations that need to be executed in order but might have asynchronous results, such as API calls, file processing, or any task that requires throttling.

What are queue's main functionalities?

Basic Queue Functionality

This demonstrates how to create a basic queue, add tasks to it, and start processing. Each task is a function that accepts a callback, which must be called upon completion.

const queue = require('queue');
const q = queue();
q.push(function(cb) {
  console.log('Hello');
cb();
});
q.push(function(cb) {
  console.log('World');
cb();
});
q.start(function(err) {
  console.log('All tasks finished.');
});

Concurrency Control

This example shows how to set a concurrency limit, allowing up to 2 tasks to be processed simultaneously.

const q = queue({concurrency: 2});
// Add tasks to q
q.start(function(err) {
  console.log('All tasks processed with a maximum of 2 tasks concurrently.');
});

Timeout for Tasks

This code sets a timeout for each task in the queue. If a task does not call its callback within the specified timeout, the queue will move on to the next task.

const q = queue();
q.timeout = 1000; // 1 second timeout for each task
q.push(function(cb) {
  setTimeout(function() {
    console.log('This task will timeout');
    cb();
  }, 1500); // This task takes longer than the timeout
});
q.start();

Other packages similar to queue

async

The 'async' package provides a wide array of functionalities for working with asynchronous JavaScript, including queue management. Compared to 'queue', 'async' offers more comprehensive control over asynchronous flow control but might be more complex for simple queue needs.

bull

Bull is a Redis-backed queue package for handling distributed jobs and messages in Node.js. It's more suited for scenarios requiring robustness, such as background processing or job scheduling, and offers features like prioritization, repeatable jobs, and event listeners. It's more complex and feature-rich compared to 'queue', which is simpler and doesn't require Redis.

p-queue

p-queue is a promise-based queue with concurrency control, similar to 'queue' but leveraging Promises for task handling. It provides an easy-to-use API for managing asynchronous tasks with more modern JavaScript syntax. It's a good alternative if you prefer working with Promises over callbacks.

README

   ____  __  _____  __  _____
  / __ `/ / / / _ \/ / / / _ \
 / /_/ / /_/ /  __/ /_/ /  __/
 \__, /\__,_/\___/\__,_/\___/
   /_/

Asynchronous function queue with adjustable concurrency.

This module exports a class Queue that implements most of the Array API. Pass async functions (ones that accept a callback or return a promise) to an instance's additive array methods. Processing begins when you call q.start().

Example

npm run example

var queue = require('../')

var q = queue()
var results = []

// add jobs using the familiar Array API
q.push(function (cb) {
  results.push('two')
  cb()
})

q.push(
  function (cb) {
    results.push('four')
    cb()
  },
  function (cb) {
    results.push('five')
    cb()
  }
)

// jobs can accept a callback or return a promise
q.push(function () {
  return new Promise(function (resolve, reject) {
    results.push('one')
    resolve()
  })
})

q.unshift(function (cb) {
  results.push('one')
  cb()
})

q.splice(2, 0, function (cb) {
  results.push('three')
  cb()
})

// use the timeout feature to deal with jobs that
// take too long or forget to execute a callback
q.timeout = 100

q.on('timeout', function (next, job) {
  console.log('job timed out:', job.toString().replace(/\n/g, ''))
  next()
})

q.push(function (cb) {
  setTimeout(function () {
    console.log('slow job finished')
    cb()
  }, 200)
})

q.push(function (cb) {
  console.log('forgot to execute callback')
})

// get notified when jobs complete
q.on('success', function (result, job) {
  console.log('job finished processing:', job.toString().replace(/\n/g, ''))
})

// begin processing, get notified on end / failure
q.start(function (err) {
  if (err) throw err
  console.log('all done:', results)
})

Install

npm install queue

Test

npm test npm run test-browser

API

var q = queue([opts])

Constructor. opts may contain inital values for:

• q.concurrency
• q.timeout
• q.autostart
• q.results

Instance methods

q.start([cb])

cb, if passed, will be called when the queue empties or when an error occurs.

q.stop()

Stops the queue. can be resumed with q.start().

q.end([err])

Stop and empty the queue immediately.

Instance methods mixed in from Array

Mozilla has docs on how these methods work here. Note that slice does not copy the queue.

q.push(element1, ..., elementN)

q.unshift(element1, ..., elementN)

q.splice(index , howMany[, element1[, ...[, elementN]]])

q.pop()

q.shift()

q.slice(begin[, end])

q.reverse()

q.indexOf(searchElement[, fromIndex])

q.lastIndexOf(searchElement[, fromIndex])

Properties

q.concurrency

Max number of jobs the queue should process concurrently, defaults to Infinity.

q.timeout

Milliseconds to wait for a job to execute its callback.

q.autostart

Ensures the queue is always running if jobs are available. Useful in situations where you are using a queue only for concurrency control.

q.results

An array to set job callback arguments on.

q.length

Jobs pending + jobs to process (readonly).

Events

q.emit('success', result, job)

After a job executes its callback.

q.emit('error', err, job)

After a job passes an error to its callback.

q.emit('timeout', continue, job)

After q.timeout milliseconds have elapsed and a job has not executed its callback.

q.emit('end'[, err])

After all jobs have been processed

Releases

The latest stable release is published to npm. Abbreviated changelog below:

• 4.4
  • Add results feature
• 4.3
  • Add promise support (@kwolfy)
• 4.2
  • Unref timers on end
• 4.1
  • Add autostart feature
• 4.0
  • Change license to MIT
• 3.1.x
  • Add .npmignore
• 3.0.x
  • Change the default concurrency to Infinity
  • Allow q.start() to accept an optional callback executed on q.emit('end')
• 2.x
  • Major api changes / not backwards compatible with 1.x
• 1.x
  • Early prototype

License

Copyright © 2014 Jesse Tane [e-mail]

This work is free. You can redistribute it and/or modify it under the terms of the MIT License. See LICENSE for full details.