oxen-queue

CAUTION: This project isn't yet ready for use by humans.

Tests:

Oxen Queue

A no-frills, high-throughput worker queue backed by MySQL.

Features:

• Job persistence
• Job priority
• Job deduplication
• Concurrency
• Delayed jobs
• Multi-process/server operation

Motivation

Oxen is designed to help you chew through a very high number of jobs by leveraging significant concurrency. It is resilient to misbehaving jobs, dropped database connections, and other ills. At Opteo, we mostly use it to work though scheduled batch tasks that aren't reasonable to run in a fat Promise.all().

There are already several great job queue libraries out there, but in the context of our use-cases, they either struggled with a high number of jobs, handled unexpected disconnections poorly, or had issues with race conditions.

You'll be happy with Oxen if you:

• Have many, many jobs (millions per day isn't unreasonable)
• You're more interested in throughput than latency when it comes to job completion
• You want to be able to run arbitrary queries on the queue using SQL
• You're already running MySQL, and you don't want to add a another database to your stack (eg. Kafka)

Oxen isn't for you if:

• You need retry mechanisms for failed jobs
• Your jobs are user-facing and need to start in sub-second latencies
• You need a UI, and you don't want to hack something together yourself
• Using MySQL for a queue makes you feel icky

Installation

Infrastructure Requirements:

• Node 7 or higher
• MySQL

NPM

To install via npm, run:

npm install oxen-queue

Usage

Initialisation

Here's how you initialise the queue.

const oxen_queue = require('oxen-queue')

const ox = new oxen_queue({
    mysql_config: {
        // this object gets passed onto the mysql2 library
        user: 'mysql_user',
        password: 'mysql_password',
        // anything else you need to pass to the mysql lib
    },
    db_table: 'oxen_queue', // (optional) name the table that oxen will use
    job_type: 'avatar_renders', // give this queue a job type. Other instances of oxen with the same job type will be the same queue.
})


/* If this is your first time running oxen, run this line to automatically create the database table. You should only need to run this once. */
await ox.createTable()

Jobs

Each job is saved as a single row in your table. The actual job body is JSON.stringify'ed and put into a VARCHAR(1000) field, so anything that will survive that process will fit into a job. If 1000 characters isn't enough for you, feel free to alter your table to use a TEXT field.

const oxen_queue = require('oxen-queue')

const ox = new oxen_queue({ /* Initialisation args here */ }}

// adding a job with a string body
ox.addJob({
    body : 'job_body_here'
})

// adding a job with an object body
ox.addJob({
    body : { oh : 'hello', arr : [1, 2]}
})

// shorthand for adding a job with no additional parameters
ox.addJob('job_body_here')

TODO: define all available args for addJob() and describe job consumer.