qjobs

What is qjobs?

The qjobs npm package is a lightweight, fast job queue manager for Node.js. It allows for efficient management of tasks with the ability to limit the number of concurrent jobs, making it suitable for scenarios where resource management and job scheduling are critical. It's particularly useful for managing CPU-intensive tasks or IO-bound operations in a controlled manner.

What are qjobs's main functionalities?

Job scheduling and concurrency control

This feature allows you to schedule jobs and control the number of jobs running concurrently. In the code sample, a new QJobs instance is created with a maximum concurrency of 5, meaning no more than 5 jobs will run at the same time. Jobs are added to the queue and executed with controlled concurrency.

const QJobs = require('qjobs');

const myQueue = new QJobs({maxConcurrency: 5});

const myJob = function(args, next) {
    console.log('Working on job with args', args);
    next();
};

for (let i = 0; i < 10; i++) {
    myQueue.add(myJob, [i]);
}

myQueue.run();

Job completion and error handling

This feature demonstrates how to handle job completion and errors. Jobs are defined with a condition to either complete successfully or fail. The QJobs instance listens for 'end' and 'error' events to handle job completion and errors respectively. This allows for robust error handling and notification upon job queue completion.

const QJobs = require('qjobs');

const myQueue = new QJobs();

const myJob = function(args, next) {
    if (args.shouldFail) {
        throw new Error('Job failed');
    } else {
        console.log('Job completed successfully');
        next();
    }
};

myQueue.on('end', function() {
    console.log('All jobs have been processed');
});

myQueue.on('error', function(err) {
    console.error('Error processing job:', err);
});

myQueue.add(myJob, [{shouldFail: false}]);
myQueue.add(myJob, [{shouldFail: true}]);

myQueue.run();

Other packages similar to qjobs

bull

Bull is a more feature-rich job and message queue package for Node.js, built on top of Redis. It offers advanced functionalities such as delayed jobs, repeatable jobs, job prioritization, and more. Compared to qjobs, Bull is better suited for applications requiring complex job management features and persistence.

kue

Kue is a priority job queue backed by Redis, focusing on job creation, processing, and management. It provides a rich set of features like job events, job prioritization, and a UI for monitoring. While Kue offers more advanced features compared to qjobs, it's also heavier and may require more setup due to its Redis dependency.

bee-queue

Bee-queue is a simple, fast, and robust job/task queue for Node.js, also backed by Redis. It's designed for high-performance and reliability, supporting job progress, retries, and failures. Bee-queue is a good middle ground between qjobs and more complex solutions like Bull, offering a balance of simplicity and functionality.

README

qjobs

Efficient queue job manager module for nodejs.

Features

• Concurrency limiter
• Dynamic queue, a job can be added while the queue is running
• Optional delay before continuing after max concurrency has been reached
• Support of pause/unpause
• Events emitter based: start, end, sleep, continu, jobStart, jobEnd
• Quick statistic function, so you can know where the queue is, at regular interval

For what it can be usefull ?

Jobs which needs to run in parallels, but in a controled maner, example:

• Network scanners
• Parallels monitoring jobs
• Images/Videos related jobs

Compatibility :

• not tested with nodejs < 0.10

Examples

(take a look at tests directory if you are looking for running samples)

var qjobs = new require('./qjobs');
                                
// My non blocking main job     
var myjob = function(args,next) {
    setTimeout(function() {
        console.log('Do something interesting here',args);
        next();
    },1000);
}

var q = new qjobs({maxConcurrency:10});

// Let's add 30 job to the queue
for (var i = 0; i<30; i++) {
    q.add(myjob,[i,'test '+i]);
}

q.on('start',function() {
    console.log('Starting ...');
});

q.on('end',function() {
    console.log('... All jobs done');
});

q.on('jobStart',function(args) {
    console.log('jobStart',args);
});

q.on('jobEnd',function(args) {

    console.log('jobend',args);

    // If i'm jobId 10, then make a pause of 5 sec

    if (args._jobId == 10) {
        q.pause(true);
        setTimeout(function() {
            q.pause(false);
        },5000);
    }
});

q.on('pause',function(since) {
    console.log('in pause since '+since+' milliseconds');
});

q.on('unpause',function() {
    console.log('pause end, continu ..');
});

q.run();

//q.abort() will empty jobs list