rethinkdb-job-queue

Introduction

rethinkdb-job-queue is a persistent job or task queue backed by RethinkDB. It has been built as an alternative to the many queues available on NPM.

Please Star on GitHub / NPM and Watch for updates.

Features

• Powered by RethinkDB
• Connect to multiple databases
• Create multiple queues
• Distributed worker nodes (PubSub)
• Global queue events
• Global job cancellation
• Global pause queue
• Run concurrent jobs
• Promise based with minimal callbacks
• Priority processing of jobs
• Job progress updates
• Delayed job start
• Find jobs in the queue
• Determine job uniqueness
• Job timeout
• Retrying failed jobs
• Repeatable jobs
• Job reanimation
• Job Editing
• Rich job history log
• Over 2000 integration tests

Documentation

• Full Documentation
• Change Log
• Code Coverage Report from more than 2000 Integration Tests

Quick Start

Installation

Note: You will need to install RethinkDB before you can use rethinkdb-job-queue.

npm install rethinkdb-job-queue --save

See the installation document for supported Node.js versions and workarounds.

Simple Example

const Queue = require('rethinkdb-job-queue')
const qOptions = {
  name: 'Mathematics' // The queue and table name
}
const cxnOptions = {
  db: 'JobQueue', // The name of the database in RethinkDB
}

const q = new Queue(cxnOptions, qOptions)

const job = q.createJob({
  numerator: 123,
  denominator: 456
})

q.process((job, next) => {
    try {
      let result = job.numerator / job.denominator
      // Do something with your result
      return next(null, result)
    } catch (err) {
      console.error(err)
      return next(err)
    }
})

return q.addJob(job).catch((err) => {
  console.error(err)
})

E-Mail Job Example using nodemailer

// The following is not related to rethinkdb-job-queue.
// This is the nodemailer configuration
const nodemailer = require('nodemailer')
const transporter = nodemailer.createTransport({
  service: 'Mailgun',
  auth: {
    user: '[e-mail]',
    pass: 'your-api-key-here'
  }
})

// Setup e-mail data with unicode symbols
var mailOptions = {
  from: '"Registration" <[e-mail]>', // Sender address
  subject: 'Registration', // Subject line
  text: 'Click here to complete your registration', // Plaintext body
  html: '<b>Click here to complete your registration</b>' // HTML body
}

// rethinkdb-job-queue configuration
const Queue = require('rethinkdb-job-queue')

// Queue options have defaults and are not required
const qOptions = {
  name: 'RegistrationEmail', // The queue and table name
  masterInterval: 310000, // Database review period in milliseconds
  changeFeed: true, // Enables events from the database table
  concurrency: 100,
  removeFinishedJobs: 2592000000, // true, false, or number of milliseconds
}

// Connection options have defaults and are not required
// You can replace these options with a rethinkdbdash driver object
const cxnOptions = {
  host: 'localhost',
  port: 28015,
  db: 'JobQueue', // The name of the database in RethinkDB
}

// This is the main queue instantiation call
const q = new Queue(cxnOptions, qOptions)

// Customizing the default job options for new jobs
q.jobOptions = {
  priority: 'normal',
  timeout: 300000,
  retryMax: 3, // Four attempts, first then three retries
  retryDelay: 600000 // Time in milliseconds to delay retries
}

const job = q.createJob()
// The createJob method will only create the job locally.
// It will need to be added to the queue.
// You can decorate the job with any data to be saved for processing
job.recipient = '[e-mail]'

q.process((job, next) => {
  // Send email using job.recipient as the destination address
  mailOptions.to = job.recipient
  return transporter.sendMail(mailOptions).then((info) => {
    console.dir(info)
    return next(null, info)
  }).catch((err) => {
    // This catch is for nodemailer sendMail errors.
    return next(err)
  })
})

return q.addJob(job).then((savedJobs) => {
  // savedJobs is an array of the jobs added with updated properties
}).catch((err) => {
  console.error(err)
})

About the Owner

I, Grant Carthew, am a technologist, trainer, and Dad from Queensland, Australia. I work on code in a number of personal projects and when the need arises I build my own packages.

This project exists because there were no functional job queues built on the RethinkDB database. I wanted an alternative to the other job queues on NPM.

Everything I do in open source is done in my own time and as a contribution to the open source community.

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request :D

Please see the debugging and testing documents for more detail.

Credits

Thanks to the following marvelous packages and people for their hard work:

• The RethinkDB team for the great database.
• The RethinkDB driver rethinkdbdash by Michel
• The Promise library Bluebird by Petka Antonov.
• The uuid package.

This list could go on...

License

MIT