What is queue?
The 'queue' npm package is a fast, robust, and extensible queue implementation for managing a list of tasks in a sequential manner. It allows for asynchronous task processing, concurrency control, timeout for tasks, and pausing/resuming the queue. This package is particularly useful for rate-limiting tasks or operations that need to be executed in order but might have asynchronous results, such as API calls, file processing, or any task that requires throttling.
What are queue's main functionalities?
Basic Queue Functionality
This demonstrates how to create a basic queue, add tasks to it, and start processing. Each task is a function that accepts a callback, which must be called upon completion.
const queue = require('queue');
const q = queue();
q.push(function(cb) {
console.log('Hello');
cb();
});
q.push(function(cb) {
console.log('World');
cb();
});
q.start(function(err) {
console.log('All tasks finished.');
});
Concurrency Control
This example shows how to set a concurrency limit, allowing up to 2 tasks to be processed simultaneously.
const q = queue({concurrency: 2});
// Add tasks to q
q.start(function(err) {
console.log('All tasks processed with a maximum of 2 tasks concurrently.');
});
Timeout for Tasks
This code sets a timeout for each task in the queue. If a task does not call its callback within the specified timeout, the queue will move on to the next task.
const q = queue();
q.timeout = 1000; // 1 second timeout for each task
q.push(function(cb) {
setTimeout(function() {
console.log('This task will timeout');
cb();
}, 1500); // This task takes longer than the timeout
});
q.start();
Other packages similar to queue
async
The 'async' package provides a wide array of functionalities for working with asynchronous JavaScript, including queue management. Compared to 'queue', 'async' offers more comprehensive control over asynchronous flow control but might be more complex for simple queue needs.
bull
Bull is a Redis-backed queue package for handling distributed jobs and messages in Node.js. It's more suited for scenarios requiring robustness, such as background processing or job scheduling, and offers features like prioritization, repeatable jobs, and event listeners. It's more complex and feature-rich compared to 'queue', which is simpler and doesn't require Redis.
p-queue
p-queue is a promise-based queue with concurrency control, similar to 'queue' but leveraging Promises for task handling. It provides an easy-to-use API for managing asynchronous tasks with more modern JavaScript syntax. It's a good alternative if you prefer working with Promises over callbacks.
____ _ _____ _ _____
/ _ \/ \ /\/ __// \ /\/ __/
| / \|| | ||| \ | | ||| \
| \_\|| \_/|| /_ | \_/|| /_
\____\\____/\____\\____/\____\
An async job queue with adjustable concurrency.
Why
async's queue expects you to have one worker and many jobs. This queue simply expects a list of async functions, which is a bit more flexible - otherwise it's the same idea.
How
The module exports a class named Queue
. Pass the desired concurrency to the constructor, or change it later via the concurrency
property. Pass async functions (ones that accept a callback) to an instance's push()
method. Processing begins automatically on process.nextTick()
.
Install
npm install queue
Properties
concurrency
maximum number of jobs that the queue should process concurrently - the default is 1
Methods
push(job, cb)
add a job (and optional callback) to the queueempty()
remove any remaining jobs in the queuerun()
force run the queue immediately
Events
"advance"
fires after any job finishes"drain"
fires when the queue finishes processing all its jobs
Usage
var Queue = require("../queue");
var results = [];
var q = new Queue();
q.on("drain", function () {
console.log("All done:", results);
});
q.push(function (cb) {
results.push("one");
cb();
});
q.push([
function (cb) {
results.push("two");
cb();
},
function (cb) {
results[2] = "three";
cb();
}
]);