query-swarm

Query Swarm

Query Swarm allows you to safely distribute query-driven tasks over a swarm of parallel functions on a single process or across multiple processes/machines.

tl;dr

var redis = require('redis').createClient();
var QuerySwarm = require('../lib/QuerySwarm.js')(redis);

function query(cursor, callback) {
	// you can get a starting cursor from another data source if you wish
	cursor = cursor || 0;

	// perform the query (you would probably connect to a database here...)
	var results = [], newCursor = cursor;
	while (newCursor < cursor + 100) {
		results.push(newCursor);
		newCursor++;
	}

	// return any error, the new cursor, and results
	callback(null, newCursor, results);
}

function worker(task, callback) {
	// do something with the task
	var result = task * 2;

	// return any error, and the result
	callback(null, result);
}

var swarm = new QuerySwarm('myFirstSwarm', query, worker);
swarm.start();
swarm.stop(function(){
	// this node has gracefully stopped
});

####Use when the following are important:

• Tasks should be allowed a long time to complete
• Each task must only ever be processed once
• Failed tasks must not just "disappear"
• Nodes should shutdown gracefully

####Do not use if the following are required:

• Tasks must be strictly processed in order

Requirements

Query Swarm relies on redis to store the lock, cursor, queue, and lists responsible for swarm coordination.

Usage

See tl;dr above.

Events

• error: Error, [message]
• populate: cursor, tasks
• consume: task, [result]
• deadletter: task
• acknowledge: task
• requeue: task

Caveats

Processing retry functionality via the maxProcessingRetries option, assumes your tasks are all unique at any given point in time.