levelup

What is levelup?

LevelUP is a Node.js library that provides a simple interface for interacting with LevelDB, a fast key-value storage library. It allows for efficient storage and retrieval of data, making it suitable for applications that require high-performance data operations.

What are levelup's main functionalities?

Basic Put and Get

This feature allows you to store and retrieve key-value pairs in the database. The `put` method is used to store a value with a specific key, and the `get` method is used to retrieve the value associated with a key.

const level = require('level');
const db = level('./mydb');

// Put a key-value pair
await db.put('name', 'LevelUP');

// Get the value for a key
const value = await db.get('name');
console.log(value); // 'LevelUP'

Batch Operations

Batch operations allow you to perform multiple put and delete operations in a single atomic action. This is useful for making multiple changes to the database efficiently.

const level = require('level');
const db = level('./mydb');

// Perform batch operations
await db.batch()
  .put('name', 'LevelUP')
  .put('type', 'database')
  .del('oldKey')
  .write();

Streams

Streams provide a way to read and write data in a continuous flow. The `createReadStream` method allows you to read all key-value pairs in the database as a stream, which is useful for processing large datasets.

const level = require('level');
const db = level('./mydb');

// Create a read stream
const stream = db.createReadStream();

stream.on('data', ({ key, value }) => {
  console.log(`${key} = ${value}`);
});

Sublevel

Sublevel allows you to create isolated sub-databases within a LevelDB instance. This is useful for organizing data into different namespaces.

const level = require('level');
const sublevel = require('subleveldown');
const db = level('./mydb');
const subdb = sublevel(db, 'sub');

// Put and get in sublevel
await subdb.put('name', 'SubLevelUP');
const value = await subdb.get('name');
console.log(value); // 'SubLevelUP'

Other packages similar to levelup

leveldown

LevelDOWN is a lower-level binding for LevelDB, providing a more direct interface to the LevelDB library. It is used as a backend for LevelUP but can be used independently for more fine-grained control over LevelDB operations.

rocksdb

RocksDB is a high-performance key-value store developed by Facebook. It is similar to LevelDB but offers additional features like column families and more tunable performance options. It can be used as an alternative to LevelDB for applications requiring higher performance.

redis

Redis is an in-memory key-value store known for its speed and support for various data structures like strings, hashes, lists, sets, and more. Unlike LevelDB, Redis operates entirely in memory, making it suitable for use cases where low-latency access is critical.

README

LevelUP

Fast & simple storage - a Node.js-style LevelDB wrapper

LevelDB is a simple key/value data store built by Google, inspired by BigTable. It's used in Google Chrome and many other products. LevelDB supports arbitrary byte arrays as both keys and values, singular get, put and delete operations, batched put and delete, forward and reverse iteration and simple compression using the Snappy algorithm which is optimised for speed over compression.

LevelUP aims to expose the features of LevelDB in a Node.js-friendly way. Both keys and values are treated as Buffer objects and are automatically converted using a specified 'encoding'. LevelDB's iterators are exposed as a Node.js style object-ReadStream and writing can be peformed via an object-WriteStream.

LevelUP is an OPEN Open Source Project, see the Contributing section to find out what this means.

• Tested & supported platforms
• Basic usage
• API
• Events
• JSON data
• Important considerations
• Contributing
• Licence & copyright

Tested & supported platforms

• Linux (tested on Ubuntu)
• Mac OS
• Solaris (tested on SmartOS & Nodejitsu)

Windows support is coming soon; see issue #5 if you would like to help on that front.

Basic usage

All operations are asynchronous although they don't necessarily require a callback if you don't need to know when the operation was performed.

var levelup = require('levelup')

// 1) Create our database, supply location and options.
//    This will create or open the underlying LevelDB store.
var options = { createIfMissing: true, errorIfExists: false }
var db = levelup('./mydb', options)

// 2) put a key & value
db.put('name', 'LevelUP', function (err) {
  if (err) return console.log('Ooops!', err) // some kind of I/O error

  // 3) fetch by key
  db.get('name', function (err, value) {
    if (err) return console.log('Ooops!', err) // likely the key was not found

    // ta da!
    console.log('name=' + value)
  })
})

API

• levelup()
• db.open()
• db.close()
• db.put()
• db.get()
• db.del()
• db.batch()
• db.isOpen()
• db.isClosed()
• db.readStream()
• db.keyStream()
• db.valueStream()
• db.writeStream()

---

levelup(location[, options[, callback]])

levelup() is the main entry point for creating a new LevelUP instance and opening the underlying store with LevelDB.

This function returns a new instance of LevelUP and will also initiate an open() operation. Opening the database is an asynchronous operation which will trigger your callback if you provide one. The callback should take the form: function (err, db) {} where the db is the LevelUP instance. If you don't provide a callback, any read & write operations are simply queued internally until the database is fully opened.

This leads to two alternative ways of managing a new LevelUP instance:

levelup(location, options, function (err, db) {
  if (err) throw err
  db.get('foo', function (err, value) {
    if (err) return console.log('foo does not exist')
    console.log('got foo =', value)
  })
})

// vs the equivalent:

var db = levelup(location, options) // will throw if an error occurs
db.get('foo', function (err, value) {
  if (err) return console.log('foo does not exist')
  console.log('got foo =', value)
})

options

levelup() takes an optional options object as its second argument; the following properties are accepted:

• 'createIfMissing' (boolean, default: false): If true, will initialise an empty database at the specified location if one doesn't already exit. If false and a database doesn't exist you will receive an error in your open() callback and your database won't open.

• 'errorIfExists' (boolean, default: false): If true, you will receive an error in your open() callback if the database exists at the specified location.

• 'compression' (boolean, default: true): If true, all compressible data will be run through the Snappy compression algorithm before being stored. Snappy is very fast and shouldn't gain much speed by disabling so leave this on unless you have good reason to turn it off.

• 'encoding' (string, default: 'utf8'): The encoding of the keys and values passed through Node.js' Buffer implementation (see Buffer#toString())

  'utf8' is the default encoding for both keys and values so you can simply pass in strings and expect strings from your get() operations. You can also pass Buffer objects as keys and/or values and converstion will be performed.

  Supported encodings are: hex, utf8, ascii, binary, base64, ucs2, utf16le.

  'json' encoding is also supported, see below.

• 'keyEncoding' and valueEncoding' (string, default: 'utf8'): use instead of encoding to specify the exact encoding of both the keys and the values in this database.

Additionally, each of the main interface methods accept an optional options object that can be used to override encoding (or keyEncoding & valueEncoding).

---

db.open([callback])

open() opens the underlying LevelDB store. In general you should never need to call this method directly as it's automatically called by levelup().

However, it is possible to reopen a database after it has been closed with close(); although this is not generally advised.

---

db.close([callback])

close() closes the underlying LevelDB store. The callback will receive any error encountered during closing as the first argument.

You should always clean up your LevelUP instance by calling close() when you no longer need it to free up resources. A LevelDB store cannot be opened by multiple instances of LevelDB/LevelUP simultaneously.

---

db.put(key, value[, options][, callback])

put() is the primary method for inserting data into the store. Both the key and value can be arbitrary data objects.

The callback argument is optional but if you don't provide one and an error occurs then expect the error to be thrown.

options

Encoding of the key and value objects will adhere to encoding option(s) provided to levelup(), although you can provide alternative encoding settings in the options for put() (it's recommended that stay consistent in your encoding of keys and values in a single store).

If you provide a 'sync' value of true in your options object, LevelDB will perform a synchronous write of the data; although the operation will be asynchronous as far as Node is concerned. Normally, LevelDB passes the data to the operating system for writing and returns immediately, however a synchronous write will use fsync() or equivalent so your callback won't be triggered until the data is actually on disk. Synchronous filesystem writes are significantly slower than asynchronous writes but if you want to be absolutely sure that the data is flushed then you can use 'sync': true.

---

db.get(key[, options][, callback])

get() is the primary method for fetching data from the store. The key can be an arbitrary data object but if it doesn't exist in the store then the callback will receive an error as its first argument.

options

Encoding of the key objects will adhere to encoding option(s) provided to levelup(), although you can provide alternative encoding settings in the options for et() (it's recommended that stay consistent in your encoding of keys and values in a single store).

---

db.del(key[, options][, callback])

del() is the primary method for removing data from the store. The key can be an arbitrary data object but if it doesn't exist in the store then the callback will receive an error as its first argument.

options

Encoding of the key objects will adhere to encoding option(s) provided to levelup(), although you can provide alternative encoding settings in the options for et() (it's recommended that stay consistent in your encoding of keys and values in a single store).

A 'sync' option can also be passed, see put() for details on how this works.

---

db.batch(array[, options][, callback])

batch() can be used for very fast bulk-write operations (both put and delete). The array argument should contain a list of operations to be executed sequentially. Each operation is contained in an object having the following properties: type, key, value, where the type is either 'put' or 'del'. In the case of 'del' the 'value' property is ignored.

var ops = [
    { type: 'del', key: 'father' }
  , { type: 'put', key: 'name', value: 'Yuri Irsenovich Kim' }
  , { type: 'put', key: 'dob', value: '16 February 1941' }
  , { type: 'put', key: 'spouse', value: 'Kim Young-sook' }
  , { type: 'put', key: 'occupation', value: 'Clown' }
]

db.batch(ops, function (err) {
  if (err) return console.log('Ooops!', err)
  console.log('Great success dear leader!')
})

options

See put() for a discussion on the options object. You can overwrite default key and value encodings and also specify the use of sync filesystem operations.

---

db.isOpen()

A LevelUP object can be in one of the following states:

• "new" - newly created, not opened or closed
• "opening" - waiting for the database to be opened
• "open" - successfully opened the database, available for use
• "closing" - waiting for the database to be closed
• "closed" - database has been successfully closed, should not be used

isOpen() will return true only when the state is "open".

---

db.isClosed()

See isOpen()

isClosed() will return true only when the state is "closing" or "closed", it can be useful for determining if read and write operations are permissible.

---

db.readStream()

You can obtain a ReadStream of the full database by calling the readStream() method. The resulting stream is a complete Node.js-style Readable Stream where 'data' events emit objects with 'key' and 'value' pairs.

db.readStream()
  .on('data', function (data) {
    console.log(data.key, '=', data.value)
  })
  .on('error', function (err) {
    console.log('Oh my!', err)
  })
  .on('close', function () {
    console.log('Stream closed')
  })
  .on('end', function () {
    console.log('Stream closed')
  })

The standard pause(), resume() and destroy() methods are implemented on the ReadStream, as is pipe() (see below). 'data', 'error', 'end' and 'close' events are emitted.

Additionally, you can supply an options object as the first parameter to readStream() with the following options:

• 'start': the key you wish to start the read at. By default it will start at the beginning of the store. Note that the start doesn't have to be an actual key that exists, LevelDB will simply find the next key, greater than the key you provide.

• 'end': the key you wish to end the read on. By default it will continue until the end of the store. Again, the end doesn't have to be an actual key as an (inclusive) <=-type operation is performed to detect the end. You can also use the destroy() method instead of supplying an 'end' parameter to achieve the same effect.

• 'reverse' (boolean, default: false): a boolean, set to true if you want the stream to go in reverse order. Beware that due to the way LevelDB works, a reverse seek will be slower than a forward seek.

• 'keys' (boolean, default: true): whether the 'data' event should contain keys. If set to true and 'values' set to false then 'data' events will simply be keys, rather than objects with a 'key' property. Used internally by the keyStream() method.

• 'values' (boolean, default: true): whether the 'data' event should contain values. If set to true and 'keys' set to false then 'data' events will simply be values, rather than objects with a 'value' property. Used internally by the valueStream() method.

• 'limit' (number, default: -1): limit the number of results collected by this stream. This number represents a maximum number of results and may not be reached if you get to the end of the store or your 'end' value first. A value of -1 means there is no limit.

---

db.keyStream()

A KeyStream is a ReadStream where the 'data' events are simply the keys from the database so it can be used like a traditional stream rather than an object stream.

You can obtain a KeyStream either by calling the keyStream() method on a LevelUP object or by passing passing an options object to readStream() with keys set to true and values set to false.

db.keyStream()
  .on('data', function (data) {
    console.log('key=', data)
  })

// same as:
db.readStream({ keys: true, values: false })
  .on('data', function (data) {
    console.log('key=', data)
  })

---

db.valueStream()

A ValueStream is a ReadStream where the 'data' events are simply the values from the database so it can be used like a traditional stream rather than an object stream.

You can obtain a ValueStream either by calling the valueStream() method on a LevelUP object or by passing passing an options object to readStream() with valuess set to true and keys set to false.

db.valueStream()
  .on('data', function (data) {
    console.log('value=', data)
  })

// same as:
db.readStream({ keys: false, values: true })
  .on('data', function (data) {
    console.log('value=', data)
  })

---

db.writeStream()

A WriteStream can be obtained by calling the writeStream() method. The resulting stream is a complete Node.js-style Writable Stream which accepts objects with 'key' and 'value' pairs on its write() method. Tce WriteStream will buffer writes and submit them as a batch() operation where the writes occur on the same event loop tick, otherwise they are treated as simple put() operations.

db.writeStream()
  .on('error', function (err) {
    console.log('Oh my!', err)
  })
  .on('close', function () {
    console.log('Stream closed')
  })
  .write({ key: 'name', value: 'Yuri Irsenovich Kim' })
  .write({ key: 'dob', value: '16 February 1941' })
  .write({ key: 'spouse', value: 'Kim Young-sook' })
  .write({ key: 'occupation', value: 'Clown' })
  .end()

The standard write(), end(), destroy() and destroySoon() methods are implemented on the WriteStream. 'drain', 'error', 'close' and 'pipe' events are emitted.

Additionally, you can supply an options object as the first parameter to writeStream() with the following option:

• 'useBatch' (boolean, default: true): if set to false, your WriteStream will avoid the use of batch() and use put() to write all data to the database. Since batch() is much quicker than multiple put() operations, you are advised to leave this as true unless you have a good reason to change it.

Pipes and Node Stream compatibility

A ReadStream can be piped directly to a WriteStream, allowing for easy copying of an entire database. A simple copy() operation is included in LevelUP that performs exactly this on two open databases:

function copy (srcdb, dstdb, callback) {
  srcdb.readStream().pipe(dstdb.writeStream().on('close', callback))
}

The ReadStream is also fstream-compatible which means you should be able to pipe to and from fstreams. So you can serialize and deserialize an entire database to a directory where keys are filenames and values are their contents, or even into a tar file using node-tar. See the fstream functional test for an example. (Note: I'm not really sure there's a great use-case for this but it's a fun example and it helps to harden the stream implementations.)

KeyStreams and ValueStreams can be treated like standard streams of raw data. If 'encoding' is set to 'binary' the 'data' events will simply be standard Node Buffer objects straight out of the data store.

Events

LevelUP emits events when the callbacks to the corresponding methods are called.

• db.emit('put', key, value) emitted when a new value is 'put'
• db.emit('del', key) emitted when a value is deleted
• db.emit('batch', ary) emitted when a batch operation has executed
• db.emit('ready') emitted when the database has opened
• db.emit('closed') emitted when the database has closed

If you do not pass a callback to an async function, and there is an error, LevelUP will emit('error', err) instead.

JSON data

You specify 'json' encoding for both keys and/or values, you can then supply JavaScript objects to LevelUP and receive them from all fetch operations, including ReadStreams. LevelUP will automatically stringify your objects and store them as utf8 and parse the strings back into objects before passing them back to you.

Important considerations

• LevelDB is thread-safe but is not suitable for accessing with multiple processes. You should only ever have a LevelDB database open from a single Node.js process.

Contributing

LevelUP is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the CONTRIBUTING.md file for more details.

Contributors

• Rod Vagg - GitHub/rvagg - Twitter/@rvagg
• John Chesley - GitHub/chesles - Twitter/@chesles
• Jake Verbaten - GitHub/raynos - Twitter/@raynos2
• Dominic Tarr - GitHub/dominictarr - Twitter/@dominictarr
• Max Ogden - GitHub/maxogden - Twitter/@maxogden

Licence & copyright

LevelUP is Copyright (c) 2012 Rod Vagg and other contributors listed above.

LevelUP is licensed under the MIT licence. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.

LevelUP builds on the excellent work of the LevelDB and Snappy teams from Google and additional contributors. LevelDB and Snappy are both issued under the New BSD Licence.

Changelog

[0.4.0] - 2012-12-17

Added

• Add SQLite3 to test suite (@rvagg)
• Add basic get() benchmarks (@rvagg)
• Add compress boolean on open() (@rvagg)

Changed

• Speed up batch() and allow non-Strings to C++ (@rvagg)
• Improved compression test (@rvagg)
• Return Strings not Buffers from C++ when possible (@rvagg)
• Optimised encoders and decoders (@rvagg)
• Revamped benchmark suite (@rvagg)
• Allow JS Strings through to native layer (@rvagg)
• Cleaner build for osx (@rvagg)
• Upgrade to LevelDB@1.7 (@rvagg)

Removed

• Remove old and unused util functions (@rvagg)
• Remove compile warnings on osx (@rvagg)
• Remove compile warnings for solaris (@rvagg)

Fixed

• Fix batch() benchmarks (@rvagg)