CAUTION: This project isn't yet ready for use by humans.
Tests:
Oxen Queue
A no-frills, high-throughput worker queue backed by MySQL.
Features:
- Job persistence
- Job priority
- Job deduplication
- Concurrency
- Delayed jobs
- Multi-process/server operation
Motivation
Oxen is designed to help you chew through a very high number of jobs by leveraging significant concurrency. It is resilient to misbehaving jobs, dropped database connections, and other ills. At Opteo, we mostly use it to work though scheduled batch tasks that aren't reasonable to run in a fat Promise.all().
There are already several great job queue libraries out there, but in the context of our use-cases, they either struggled with a high number of jobs, handled unexpected disconnections poorly, or had issues with race conditions.
You'll be happy with Oxen if you:
- Have many, many jobs (millions per day isn't unreasonable)
- You're more interested in throughput than latency when it comes to job completion
- You want to be able to run arbitrary queries on the queue using SQL
- You're already running MySQL, and you don't want to add a another database to your stack (eg. Kafka)
Oxen isn't for you if:
- You need retry mechanisms for failed jobs
- Your jobs are user-facing and need to start in sub-second latencies
- You need a UI, and you don't want to hack something together yourself
- Using MySQL for a queue makes you feel icky
Installation
Infrastructure Requirements:
NPM
To install via npm, run:
npm install oxen-queue
Usage
Initialisation
Here's how you initialise the queue.
const oxen_queue = require('oxen-queue')
const ox = new oxen_queue({
mysql_config: {
user: 'mysql_user',
password: 'mysql_password',
},
db_table: 'oxen_queue',
job_type: 'avatar_renders',
})
await ox.createTable()
Jobs
Each job is saved as a single row in your table. The actual job body is JSON.stringify'ed and put into a VARCHAR(1000) field, so anything that will survive that process will fit into a job. If 1000 characters isn't enough for you, feel free to alter your table to use a TEXT field.
const oxen_queue = require('oxen-queue')
const ox = new oxen_queue({ }}
ox.addJob({
body : 'job_body_here'
})
ox.addJob({
body : { oh : 'hello', arr : [1, 2]}
})
ox.addJob('job_body_here')
TODO: define all available args for addJob() and describe job consumer.