rethinkdb-job-queue

Introduction

rethinkdb-job-queue is a persistent job or task queue backed by RethinkDB. It has been built as an alternative to the many queues available on NPM.

Please Star on GitHub / NPM and Watch for updates.

Features

• Powered by RethinkDB
• Connect to multiple databases
• Create multiple queues
• Distributed worker nodes (PubSub)
• Global queue events
• Global job cancellation
• Global pause queue
• Run concurrent jobs
• Promise based with minimal callbacks
• Priority processing of jobs
• Job progress updates
• Delayed job start
• Find jobs in the queue
• Determine job uniqueness
• Job timeout
• Retrying failed jobs
• Job reanimation
• Job Editing
• Rich job history log
• Over 1300 integration tests

Documentation

For full documentation please see the wiki

Project Status

• This rethinkdb-job-queue module is fully functional.
• There are over 1300 integration tests.
• In a few months I will bump the version up to 1.0.0 to support SemVer.
• Please provide feedback or raise issues prior to the version bump.

Quick Start

Installation

Note: You will need to install RethinkDB before you can use rethinkdb-job-queue

After installing RethinkDB install the job queue using the following command.

npm install rethinkdb-job-queue --save

Simple Example

const Queue = require('rethinkdb-job-queue')
const qOptions = {
  name: 'Mathematics' // The queue and table name
}
const cxnOptions = {
  db: 'JobQueue', // The name of the database in RethinkDB
}

const q = new Queue(cxnOptions, qOptions)

const job = q.createJob({
  numerator: 123,
  denominator: 456
})

q.process((job, next) => {
    try {
      let result = job.numerator / job.denominator
      // Do something with your result
      return next(result)
    } catch (err) {
      console.error(err)
      return next(err)
    }
})

return q.addJob(job).catch((err) => {
  console.error(err)
})

E-Mail Job Example using nodemailer

// The following is not related to rethinkdb-job-queue.
// This is the nodemailer configuration
const nodemailer = require('nodemailer')
const transporter = nodemailer.createTransport({
  service: 'Mailgun',
  auth: {
    user: '[e-mail]',
    pass: 'your-api-key-here'
  }
})

// Setup e-mail data with unicode symbols
var mailOptions = {
  from: '"Registration" <[e-mail]>', // Sender address
  subject: 'Registration', // Subject line
  text: 'Click here to complete your registration', // Plaintext body
  html: '<b>Click here to complete your registration</b>' // HTML body
}

// rethinkdb-job-queue configuration
const Queue = require('rethinkdb-job-queue')

// Queue options have defaults and are not required
const qOptions = {
  name: 'RegistrationEmail', // The queue and table name
  masterInterval: 310000, // Database review period in milliseconds
  changeFeed: true, // Enables events from the database table
  concurrency: 100,
  removeFinishedJobs: 2592000000, // true, false, or number of milliseconds
}

// Connection options have defaults and are not required
// You can replace these options with a rethinkdbdash driver object
const cxnOptions = {
  host: 'localhost',
  port: 28015,
  db: 'JobQueue', // The name of the database in RethinkDB
}

// This is the main queue instantiation call
const q = new Queue(cxnOptions, qOptions)

// Customizing the default job options for new jobs
q.jobOptions = {
  priority: 'normal',
  timeout: 300000,
  retryMax: 3, // Four attempts, first then three retries
  retryDelay: 600000 // Time in milliseconds to delay retries
}

const job = q.createJob()
// The createJob method will only create the job locally.
// It will need to be added to the queue.
// You can decorate the job with any data to be saved for processing
job.recipient = '[e-mail]'

q.process((job, next) => {
  // Send email using job.recipient as the destination address
  mailOptions.to = job.recipient
  return transporter.sendMail(mailOptions).then((info) => {
    console.dir(info)
    return next(info)
  }).catch((err) => {
    // This catch is for nodemailer sendMail errors.
    return next(err)
  })
})

return q.addJob(job).then((savedJobs) => {
  // savedJobs is an array of the jobs added with updated properties
}).catch((err) => {
  console.error(err)
})

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request :D

Please see the debugging and testing documents for more detail.

Credits

Thanks to the following marvelous packages and people for their hard work:

• The RethinkDB team for the great database.
• The RethinkDB driver rethinkdbdash by Michel
• The Promise library Bluebird by Petka Antonov.
• The uuid package.

This list could go on...

License

MIT