scurry

scurry

A leveldb-backed consistent hash ring, for your caching needs. I can see the day when you might want to put data in this & feel reasonably sort of confident you might get it back out again. If this scares you, it's supposed to.

One part each

Rod Vagg's levelup leveldb bindings for node + sublevel to create buckets.

Dominic Tarr's crtd, which uses his scuttlebutt implementation to keep a document in sync.

light-cycle, a lightweight consistent hash ring structure that can be mixed into most anything.

restify to provide a simple RESTful api to data in the buckets.

Shake with ice

Run a server:

node index.js --id=node-one -m -p 3333 -g 4114 -d ./db | ./node_modules/.bin/bunyan -o short

Run a client or five:

node index.js --id=node-two -p 3334 -g 4114 -s 10.0.0.5 -d ./db2 | ./node_modules/.bin/bunyan -o short
node index.js --id=node-three -p 3335 -g 4114 -s 10.0.0.5 -d ./db3 | ./node_modules/.bin/bunyan -o short
node index.js --id=node-four -p 3336 -g 4114 -s 10.0.0.5 -d ./db4 | ./node_modules/.bin/bunyan -o short

Replace 10.0.0.5 with the IP address of your server.

Strain into a chilled glass

Then stuff some data in:

http PUT 10.0.0.5:3334/vodkas/1 name="Sobieski" rating=5
http PUT 10.0.0.5:3335/vodkas/2 name="Tito's Handmade" rating=5
http PUT 10.0.0.5:3335/vodkas/3 name="Bimber" rating=4

Get it back out: http GET 10.0.0.5:3336/vodkas/2 | json

(Human-friendly shell commands courtesy of httpie).

Garnish with a twist of orange peel

API endpoints exposed:

• GET /:bucket: stream sorted keys for a bucket (works!)
• POST /:bucket: add an item to the cache; id is generated for you & returned
• PUT /:bucket/:id: add/update an item in the cache; 204 response
• GET /:bucket/:id: get an item from the cache
• HEAD /:bucket/:id: headers for an item
• DEL /:bucket/:id: remove an item

TTLs

Send this header to specify a time-to-live for your cached data: X-Scurry-TTL: [seconds]

Conditional requests

Scurry sends an ETag header and a last-modified timestamp.

Storage format

As of version 0.0.4, the data stored in the LevelDB nodes is json structured as follows:

{
	version:        1,                       // storage version
    ts:             Date.now(),              // timestamp of last set()
    payload:        value,                   // base64-encoded string if buffer, JSON string if not
    etag:           crc.digest('hex'),       // md5 hex digest of payload
    base64:         valueIsB64String,        // true if the payload had to be base64 encoded
    'content-type': metadata['content-type'] // content-type if passed in
};

TODO

Upcoming releases:

• Release 0.0.4 will probably make streaming keys work, maybe. Done!
• Release 0.0.5 will finalize the storage format. Probably done!
• Release 0.0.6 will contemplate eviction. Not yet!

General goals:

• Implement key streaming from multiple nodes. See notes in endpoints.handleGetBucket().
• The RESTful server needs error handling.
• Reconnect on errors.
• Error handling.
• Better logging. Configurable, for one thing.
• Back ends should be pluggable; the API is very small.
• Stretch goal: replication?

License

MIT.