d3-queue

What is d3-queue?

The d3-queue npm package is a utility for managing asynchronous tasks with a focus on concurrency control. It allows you to queue up tasks and control how many of them run simultaneously, making it easier to handle multiple asynchronous operations without overwhelming the system.

What are d3-queue's main functionalities?

Queue Creation

This feature allows you to create a new queue instance. The queue can then be used to manage multiple asynchronous tasks.

const queue = require('d3-queue').queue;
const q = queue();

Adding Tasks

This feature allows you to add tasks to the queue. Each task is a function that takes a callback as its argument. The callback should be called when the task is complete.

q.defer((callback) => {
  setTimeout(() => {
    console.log('Task 1 complete');
    callback(null, 'Task 1 result');
  }, 1000);
});

q.defer((callback) => {
  setTimeout(() => {
    console.log('Task 2 complete');
    callback(null, 'Task 2 result');
  }, 500);
});

Concurrency Control

This feature allows you to control the number of tasks that can run concurrently. By default, the concurrency is set to 1, but you can specify a different number when creating the queue.

const q = queue(2); // Allow up to 2 tasks to run concurrently

Awaiting Completion

This feature allows you to specify a callback that will be called when all tasks in the queue have completed. The callback receives any error that occurred and an array of results from the tasks.

q.awaitAll((error, results) => {
  if (error) throw error;
  console.log('All tasks complete', results);
});

Other packages similar to d3-queue

async

The async package provides a wide range of utilities for working with asynchronous JavaScript. It includes functions for parallel and series execution, as well as queue management. Compared to d3-queue, async offers a broader set of features but can be more complex to use.

p-queue

The p-queue package is a promise-based queue implementation for managing asynchronous tasks. It supports concurrency control and task prioritization. Unlike d3-queue, which uses callbacks, p-queue is designed to work with promises, making it more suitable for modern JavaScript applications.

queue

The queue package is a simple and lightweight queue implementation for managing asynchronous tasks. It supports concurrency control and task retries. While it offers similar functionality to d3-queue, it is more minimalistic and may lack some of the advanced features found in other packages.

README

d3-queue

A queue evaluates zero or more deferred asynchronous tasks with configurable concurrency: you control how many tasks run at the same time. When all the tasks complete, or an error occurs, the queue passes the results to your await callback. This library is similar to Async.js’s parallel (when concurrency is infinite), series (when concurrency is 1), and queue, but features a much smaller footprint: as of release 1.2, d3-queue is about 600 bytes gzipped, compared to 4,300 for Async.

Each task is defined as a function that takes a callback as its last argument. For example, here’s a task that says hello after a short delay:

function delayedHello(callback) {
  setTimeout(function() {
    console.log("Hello!");
    callback(null);
  }, 250);
}

When a task completes, it must call the provided callback. The first argument to the callback should be null if the task is successfull, or the error if the task failed. The optional second argument to the callback is the return value of the task. (To return multiple values from a single callback, wrap the results in an object or array.)

To run multiple tasks simultaneously, create a queue, defer your tasks, and then register an await callback to be called when all of the tasks complete (or an error occurs):

var q = queue();
q.defer(delayedHello);
q.defer(delayedHello);
q.await(function(error) {
  if (error) throw error;
  console.log("Goodbye!");
});

Of course, you can also use a for loop to defer many tasks:

var q = queue();

for (var i = 0; i < 1000; ++i) {
  q.defer(delayedHello);
}

q.awaitAll(function(error) {
  if (error) throw error;
  console.log("Goodbye!");
});

Tasks can take optional arguments. For example, here’s how to configure the delay before hello and provide a name:

function delayedHello(name, delay, callback) {
  setTimeout(function() {
    console.log("Hello, " + name + "!");
    callback(null);
  }, delay);
}

Any additional arguments provided to queue.defer are automatically passed along to the task function before the callback argument. You can also use method chaining for conciseness, avoiding the need for a local variable:

queue()
    .defer(delayedHello, "Alice", 250)
    .defer(delayedHello, "Bob", 500)
    .defer(delayedHello, "Carol", 750)
    .await(function(error) {
      if (error) throw error;
      console.log("Goodbye!");
    });

The asynchronous callback pattern is very common in Node.js, so Queue works directly with many Node APIs. For example, to stat two files concurrently:

queue()
    .defer(fs.stat, __dirname + "/../Makefile")
    .defer(fs.stat, __dirname + "/../package.json")
    .await(function(error, file1, file2) {
      if (error) throw error;
      console.log(file1, file2);
    });

You can also make abortable tasks: these tasks return an object with an abort method which terminates the task. So, if a task calls setTimeout on start, it can call clearTimeout on abort. For example:

function delayedHello(name, delay, callback) {
  var id = setTimeout(function() {
    console.log("Hello, " + name + "!");
    callback(null);
  }, delay);
  return {
    abort: function() {
      clearTimeout(id);
    }
  };
}

When you call queue.abort, any in-progress tasks will be immediately aborted; in addition, any pending (not-yet-started) tasks not be started. Note that you can also use queue.abort without abortable tasks, in which case pending tasks will be cancelled, though active tasks will continue to run. Conveniently, the d3-request library implements abort atop XMLHttpRequest. For example:

var q = queue()
    .defer(d3.request, "http://www.google.com:81")
    .defer(d3.request, "http://www.google.com:81")
    .defer(d3.request, "http://www.google.com:81")
    .awaitAll(function(error, results) {
      if (error) throw error;
      console.log(results);
    });

To abort these requests, call q.abort().

Installation

If you use NPM, npm install d3-queue. Otherwise, download the latest release. The released bundle supports AMD, CommonJS, and vanilla environments. You can also load directly from d3js.org:

<script src="https://d3js.org/queue.v1.min.js"></script>

In a vanilla environment, a queue global function is exported. Try d3-queue in your browser.

API Reference

# queue([concurrency])

Constructs a new queue with the specified concurrency. If concurrency is not specified, the queue has infinite concurrency. Otherwise, concurrency is a positive integer. For example, if concurrency is 1, then all tasks will be run in series. If concurrency is 3, then at most three tasks will be allowed to proceed concurrently; this is useful, for example, when loading resources in a web browser.

# queue.defer(task[, arguments…])

Adds the specified asynchronous task callback to the queue, with any optional arguments. The task is a function that will be called when the task should start. It is passed the specified optional arguments and an additional callback as the last argument; the callback must be invoked by the task when it finishes. The task must invoke the callback with two arguments: the error, if any, and the result of the task. To return multiple results from a single callback, wrap the results in an object or array.

For example, here’s a task which computes the answer to the ultimate question of life, the universe, and everything after a short delay:

function simpleTask(callback) {
  setTimeout(function() {
    callback(null, {answer: 42});
  }, 250);
}

If the task calls back with an error, any tasks that were scheduled but not yet started will not run. For a serial queue (of concurrency 1), this means that a task will only run if all previous tasks succeed. For a queue with higher concurrency, only the first error that occurs is reported to the await callback, and tasks that were started before the error occurred will continue to run; note, however, that their results will not be reported to the await callback.

Tasks can only be deferred before queue.await or queue.awaitAll is called. If a task is deferred after then, an error is thrown. If the task is not a function, an error is thrown.

# queue.abort()

Aborts any active tasks, invoking each active task’s task.abort function, if any. Also prevents any new tasks from starting, and invokes the queue.await or queue.awaitAll callback with an error indicating that the queue was aborted. See the introduction for an example implementation of an abortable task.

# queue.await(callback)

Sets the callback to be invoked when all deferred tasks have finished. The first argument to the callback is the first error that occurred, or null if no error occurred. If an error occurred, there are no additional arguments to the callback. Otherwise, the callback is passed each result as an additional argument. For example:

queue()
    .defer(fs.stat, __dirname + "/../Makefile")
    .defer(fs.stat, __dirname + "/../package.json")
    .await(function(error, file1, file2) { console.log(file1, file2); });

If all deferred tasks have already completed, the callback will be invoked immediately. This method may only be called once, after any tasks have been deferred. If this method is called multiple times, or if it is called after queue.awaitAll, an error is thrown. If the callback is not a function, an error is thrown.

# queue.awaitAll(callback)

Sets the callback to be invoked when all deferred tasks have finished. The first argument to the callback is the first error that occurred, or null if no error occurred. If an error occurred, there are no additional arguments to the callback. Otherwise, the callback is also passed an array of results as the second argument. For example:

queue()
    .defer(fs.stat, __dirname + "/../Makefile")
    .defer(fs.stat, __dirname + "/../package.json")
    .awaitAll(function(error, files) { console.log(files); });

If all deferred tasks have already completed, the callback will be invoked immediately. This method may only be called once, after any tasks have been deferred. If this method is called multiple times, or if it is called after queue.await, an error is thrown. If the callback is not a function, an error is thrown.