remo

ReMo

Redis and Mongo for async tasks or slow servers.

Metrics

Queue Query List

Create a list on memory, save on disk.

Standalone ReMo

Use ReMo to manage input queries.

ReMo

Configure your queues:

# conf.json
"input"  : "db:mongo:input"   # All input queries
"retry"  : "db:mongo:retry"   # Queries not processed
"output" : "db:mongo:output"  # Output from mongo[unused]
"errors" : "db:mongo:errors"  # Logging errors
"results": "db:mongo:results" # Logging results

Configure your defaults

# conf.json
{
  "defaults": {
    ...
    "url": "mongodb://127.0.0.1/remo"
    ...
  }
}

Sample ReMo Work

var remo = require('remo');
var subR = redis.createClient();
var mongo = require('mongodb').MongoClient;

mongo.connect(conf.defaults.url, function(err, db) {
  subR.lpop('db:mongo:input', function(err, message) {
    remo.insert(db, message, function(err, results) {
      subR.lpush(['db:mongo:errors', JSON.stringify(err)], empty);
      subR.lpush('db:mongo:results', JSON.stringify(results)], empty);
    });
  });
});

Learn By Example

Run your mongod v3 service

mongod --directoryperdb --storageEngine wiredTiger

Clean redis

redis-cli
127.0.0.1:6379>FLUSHALL

Flood redis

node lib/fill.js
#done!

Count input queries

redis-cli
127.0.0.1:6379>LLEN "db:mongo:input"
(integer) 100000

Open your monitor

redis-cli monitor

From redis to mongodb listening for changes

node lib/init.js

Play with times

#conf.json
{
"sleep": 1000,  # Listen for changes every second
"queue":{
  "size": 1000, # Slice whole process into blocks of 1000 queries
  "wait": 0,    # Time between each query
...
}

Flood redis again and see monitor

node lib/fill.js
#done!

{ rss: 81006592, heapTotal: 61790464, heapUsed: 38060592 }
Attempt with  1000  queries on  1430927005463

Mesure of processed queries

127.0.0.1:6379> LLEN "db:mongo:results"
(integer) 100000
127.0.0.1:6379> LRANGE "db:mongo:results" 0 1
1) "{\"ok\":1,\"n\":1}"
2) "{\"ok\":1,\"n\":1}"

## Redis List Expected Message

Save your messages on redis

var redis = require('redis');
var conf = require('../conf.json');
var pub = redis.createClient();

var message = conf.defaults;
message.doc = {version: 1, module: 'readme'};
message.action = 'insert';
pub.rpush(['db:mongo:input', JSON.stringify(message)], function(){});

Workflow

• Send JSON http request.
• Save JSON in redis.
• Listen for changes node/ruby
• Count pending queries
• Open a connection with mongo
• Process a block of queries
• Save on disk
• Close the mongo connection

Test

npm install gulp
npm install gulp-jshint

gulp

ToDo

• Remove code complexity.
• Separate by function. [OK]
• Export all in one file. [OK]
• Add inc, set, unset, upsert functions. [Ok]
• Use lower case for naming. [OK]
• Remove prototype functions. [OK]