bullmq

What is bullmq?

BullMQ is a powerful, fast, and feature-rich library for handling distributed job queues in Node.js. It is built on top of Redis and provides a robust set of features for managing background jobs, scheduling tasks, and handling concurrency.

What are bullmq's main functionalities?

Job Queue Management

BullMQ allows you to create and manage job queues. You can add jobs to a queue and process them using workers. This is useful for offloading tasks that can be processed asynchronously.

const { Queue, Worker } = require('bullmq');
const myQueue = new Queue('my-queue');

// Adding a job to the queue
myQueue.add('my-job', { foo: 'bar' });

// Processing jobs from the queue
const worker = new Worker('my-queue', async job => {
  console.log(job.name); // 'my-job'
  console.log(job.data); // { foo: 'bar' }
});

Job Scheduling

BullMQ supports job scheduling, allowing you to delay the execution of jobs. This is useful for tasks that need to be executed at a later time.

const { Queue } = require('bullmq');
const myQueue = new Queue('my-queue');

// Scheduling a job to run in 5 seconds
myQueue.add('my-delayed-job', { foo: 'bar' }, { delay: 5000 });

Concurrency Control

BullMQ allows you to control the concurrency of job processing. You can specify the number of jobs that can be processed simultaneously by a worker.

const { Worker } = require('bullmq');

// Creating a worker with concurrency of 5
const worker = new Worker('my-queue', async job => {
  console.log(job.name);
}, { concurrency: 5 });

Job Retries

BullMQ provides options for retrying failed jobs. You can specify the number of retry attempts and the backoff time between retries.

const { Queue } = require('bullmq');
const myQueue = new Queue('my-queue');

// Adding a job with retry options
myQueue.add('my-retry-job', { foo: 'bar' }, { attempts: 3, backoff: 5000 });

Other packages similar to bullmq

kue

Kue is a priority job queue backed by Redis, built for Node.js. It provides a simple API for creating and processing jobs, but it lacks some of the advanced features and performance optimizations found in BullMQ.

agenda

Agenda is a lightweight job scheduling library for Node.js. It uses MongoDB for persistence and provides a simple API for defining and scheduling jobs. While it is easy to use, it does not offer the same level of performance and scalability as BullMQ.

bee-queue

Bee-Queue is a simple, fast, and robust job/task queue for Node.js, backed by Redis. It is designed for high performance and low latency, but it does not have as many features as BullMQ, such as job scheduling and advanced concurrency control.

README

The fastest, most reliable, Redis-based distributed queue for Node.
Carefully written for rock solid stability and atomicity.

Read the documentation

Follow @manast for *important* Bull/BullMQ/BullMQ-Pro news and updates!

🛠 Tutorials

You can find tutorials and news in this blog: https://blog.taskforce.sh/

News 🚀

🌐 Language agnostic BullMQ

Do you need to work with BullMQ on platforms other than Node.js? If so, check out the BullMQ Proxy

🌟 Rediscover Scale Conference 2024

Discover the latest in in-memory and real-time data technologies at Rediscover Scale 2024. Ideal for engineers, architects, and technical leaders looking to push technological boundaries. Connect with experts and advance your skills at The Foundry SF, San Francisco.

Learn more and register here!

Official FrontEnd

Supercharge your queues with a professional front end:

• Get a complete overview of all your queues.
• Inspect jobs, search, retry, or promote delayed jobs.
• Metrics and statistics.
• and many more features.

Sign up at Taskforce.sh

🚀 Sponsors 🚀

Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ.

Used by

Some notable organizations using BullMQ:

The gist

Install:

$ yarn add bullmq

Add jobs to the queue:

import { Queue } from 'bullmq';

const queue = new Queue('Paint');

queue.add('cars', { color: 'blue' });

Process the jobs in your workers:

import { Worker } from 'bullmq';

const worker = new Worker('Paint', async job => {
  if (job.name === 'cars') {
    await paintCar(job.data.color);
  }
});

Listen to jobs for completion:

import { QueueEvents } from 'bullmq';

const queueEvents = new QueueEvents('Paint');

queueEvents.on('completed', ({ jobId }) => {
  console.log('done painting');
});

queueEvents.on(
  'failed',
  ({ jobId, failedReason }: { jobId: string; failedReason: string }) => {
    console.error('error painting', failedReason);
  },
);

This is just scratching the surface, check all the features and more in the official documentation

Feature Comparison

Since there are a few job queue solutions, here is a table comparing them:

Feature	BullMQ-Pro	BullMQ	Bull	Kue	Bee	Agenda
Backend	redis	redis	redis	redis	redis	mongo
Observables	✓
Group Rate Limit	✓
Group Support	✓
Batches Support	✓
Parent/Child Dependencies	✓	✓
Debouncing	✓	✓	✓
Priorities	✓	✓	✓	✓		✓
Concurrency	✓	✓	✓	✓	✓	✓
Delayed jobs	✓	✓	✓	✓		✓
Global events	✓	✓	✓	✓
Rate Limiter	✓	✓	✓
Pause/Resume	✓	✓	✓	✓
Sandboxed worker	✓	✓	✓
Repeatable jobs	✓	✓	✓			✓
Atomic ops	✓	✓	✓		✓
Persistence	✓	✓	✓	✓	✓	✓
UI	✓	✓	✓	✓		✓
Optimized for	Jobs / Messages	Jobs / Messages	Jobs / Messages	Jobs	Messages	Jobs

Contributing

Fork the repo, make some changes, submit a pull-request! Here is the contributing doc that has more details.

Thanks

Thanks for all the contributors that made this library possible, also a special mention to Leon van Kammen that kindly donated his npm bullmq repo.