burstable

Burstable

Simple and powerful task queue for Node.js on top of Beanstalkd

Features

• Wait for job completion
• Child jobs (keeping parent alive till done)

How-To

Setup

import Burstable from 'burstable';

const burstable = new Burstable(
  host, // beanstalkd host
  port, // beanstalkd port 
  {
    // Will receive job errors and job log calls
    log: function(level, err|message, meta) {
      winston.log(level, message.message || message, meta); // meta will be enhanced with tube and jobId
    }
  }
);

Spawning jobs

burstable.spawn(tube, {
  delay: 0, 
  priority: 1000,
  timeout: 10 * 60 * 1000,
  payload: {
    // job payload/values
  },
  wait: Job | [Job...] | [{tube: String, id: Number}] // Wait for jobs before handling this job - Logic on consumer side
});

Handling jobs

burstable.handle(tube, function (payload) {
  // Complete job
  return Promise.resolve();

  // Job error
  return Promise.reject();

  // Spawn child job and wait for completion before completing this job
  return this.child(anotherTube);

  // Keep someone updated
  this.log('info', 'doing the thing');
}, {
  maxTries: 3, // Total amount of tries including the first one
  backoff: {
    initial: 60 * 1000,
    exponential: 1.5 // multiple backup by N each try
  },
  // Will receive job errors and job log calls
  log: function(level, err|message, meta) {
    winston.log(level, message.message || message, meta); /// meta will be enhanced with jobId
  }
});

burstable.start(); // Enable handlers and start processing jobs, make sure handlers are setup before calling start

Dynamic tubes

Lets you dynamically update tube names for a handler. Useful if you have jobs triggered by users and you want jobs to run seqentually per user but in parallel overall (so no one blocks eachother).

burstable.handle(async function () {
  // Will be polled every 5 seconds
  let users = await db.user.findAll();

  return users.map(user => {
    return `media.process.${user.id}`
  });
}, async function (payload) {
  // do stuff
}, {
  /* all other handle options */
  /* width is disabled, use individualWidth */
});

When spawning simply construct the tubename with the appropriate values:

burstable.spawn(`media.process.${user.id}`, {
  payload: {}
});

Keep in mind that burstable will spawn a connection equal to width * amount of tubes. You'll want to make sure that your server is configured to handle that amount of connections (ulimit).

Debugging

Use DEBUG=burstable* to enable verbose debugging.