qjobs

What is qjobs?

The qjobs npm package is a lightweight, fast job queue manager for Node.js. It allows for efficient management of tasks with the ability to limit the number of concurrent jobs, making it suitable for scenarios where resource management and job scheduling are critical. It's particularly useful for managing CPU-intensive tasks or IO-bound operations in a controlled manner.

What are qjobs's main functionalities?

Job scheduling and concurrency control

This feature allows you to schedule jobs and control the number of jobs running concurrently. In the code sample, a new QJobs instance is created with a maximum concurrency of 5, meaning no more than 5 jobs will run at the same time. Jobs are added to the queue and executed with controlled concurrency.

const QJobs = require('qjobs');

const myQueue = new QJobs({maxConcurrency: 5});

const myJob = function(args, next) {
    console.log('Working on job with args', args);
    next();
};

for (let i = 0; i < 10; i++) {
    myQueue.add(myJob, [i]);
}

myQueue.run();

Job completion and error handling

This feature demonstrates how to handle job completion and errors. Jobs are defined with a condition to either complete successfully or fail. The QJobs instance listens for 'end' and 'error' events to handle job completion and errors respectively. This allows for robust error handling and notification upon job queue completion.

const QJobs = require('qjobs');

const myQueue = new QJobs();

const myJob = function(args, next) {
    if (args.shouldFail) {
        throw new Error('Job failed');
    } else {
        console.log('Job completed successfully');
        next();
    }
};

myQueue.on('end', function() {
    console.log('All jobs have been processed');
});

myQueue.on('error', function(err) {
    console.error('Error processing job:', err);
});

myQueue.add(myJob, [{shouldFail: false}]);
myQueue.add(myJob, [{shouldFail: true}]);

myQueue.run();

Other packages similar to qjobs

bull

Bull is a more feature-rich job and message queue package for Node.js, built on top of Redis. It offers advanced functionalities such as delayed jobs, repeatable jobs, job prioritization, and more. Compared to qjobs, Bull is better suited for applications requiring complex job management features and persistence.

kue

Kue is a priority job queue backed by Redis, focusing on job creation, processing, and management. It provides a rich set of features like job events, job prioritization, and a UI for monitoring. While Kue offers more advanced features compared to qjobs, it's also heavier and may require more setup due to its Redis dependency.

bee-queue

Bee-queue is a simple, fast, and robust job/task queue for Node.js, also backed by Redis. It's designed for high-performance and reliability, supporting job progress, retries, and failures. Bee-queue is a good middle ground between qjobs and more complex solutions like Bull, offering a balance of simplicity and functionality.

README

qjobs is a simple and stupid queue job manager for nodejs.

• concurrency limiter
• dynamic queue (a job can be add while the queue is treated)
• event based, can be usefull to plug within pub/sub stuffs
• non blocking (of course), but a job itself can run async code
• really simple to use
• really simple to understand

Compatibility :

• not tested with nodejs < 0.10

Example :

(take a look at tests directory for more noisy examples)

// My non blocking main job
var myjob = function(args,next) {

    // do nothing now but in 1 sec

    setTimeout(function() {

        // if i'm job id 10 or 20, let's add 
        // another job dynamicaly in the queue.
        // It can be usefull for network operation (retry on timeout) 

        if (args._jobId==10||args._jobId==20) {
            myQueueJobs.add(myjob,[999,'bla '+args._jobId]);
        }
        next();
    },1000);
}

// Notice the "new" before require, to be able to use more 
// than one queue independently
var myQueueJobs = new require('qjobs');

// Let's add 30 job and add them to the queue
for (var i = 0; i<30; i++) {
    myQueueJobs.add(myjob,[i,'test1']);
}

// I want to know when the first job has started
myQueueJobs.on('start',function() {
    console.log('starting ...');
});

// I want to know when the last job has ended
myQueueJobs.on('end',function() {
    console.log('end');
});

// I want to know when each job has started
myQueueJobs.on('jobStart',function(args) {
    console.log('jobRun',args);
});

// I want to know when each job has ended
myQueueJobs.on('jobEnd',function(args) {

    console.log('jobend',args);
    
    // If i'm jobId 10, then make a pause of 5 sec

    if (args._jobId == 10) {
        myQueueJobs.pause(true);
        setTimeout(function() {
            myQueueJobs.pause(false);
        },5000);
    }
});

// I want to know if queue is in pause every sec
myQueueJobs.on('inPause',function(since) {
    console.log('in pause since '+since+' milliseconds');
});


// JOBS !! leeeeeeeeeet's staaaaaaaart !
myQueueJobs.run();