LevelUP
Fast and simple storage. A node.js wrapper for abstract-leveldown
compliant backends.
If you are upgrading: please see CHANGELOG.md
.
Introduction
LevelDB is a simple key-value store built by Google, inspired by BigTable. It's used in Google Chrome and many other products. LevelDB supports arbitrary byte arrays as both keys and values, singular get, put and delete operations, batched put and delete, bi-directional iterators and simple compression using the very fast Snappy algorithm.
LevelUP aims to expose the features of LevelDB in a Node.js-friendly way. LevelDB's iterators are exposed as Readable Streams.
LevelDB stores entries sorted lexicographically by keys. This makes LevelUP's streaming interface a very powerful query mechanism.
LevelUP is an OPEN Open Source Project, see the Contributing section to find out what this means.
Relationship to LevelDOWN
LevelUP is designed to be backed by LevelDOWN which provides a pure C++ binding to LevelDB and can be used as a stand-alone package if required.
As of version 0.9, LevelUP no longer requires LevelDOWN as a dependency so you must npm install leveldown
when you install LevelUP.
LevelDOWN is now optional because LevelUP can be used with alternative backends, such as level.js in the browser or MemDOWN for a pure in-memory store.
LevelUP will look for LevelDOWN and throw an error if it can't find it in its Node require()
path. It will also tell you if the installed version of LevelDOWN is incompatible.
The level package is available as an alternative installation mechanism. Install it instead to automatically get both LevelUP & LevelDOWN. It exposes LevelUP on its export (i.e. you can var leveldb = require('level')
).
Tested & supported platforms
- Linux: including ARM platforms such as Raspberry Pi and Kindle!
- Mac OS
- Solaris: including Joyent's SmartOS & Nodejitsu
- Windows: Node 0.10 and above only. See installation instructions for node-gyp's dependencies here, you'll need these (free) components from Microsoft to compile and run any native Node add-on in Windows.
Basic usage
First you need to install LevelUP!
$ npm install levelup leveldown
Or
$ npm install level
(this second option requires you to use LevelUP by calling var levelup = require('level')
)
All operations are asynchronous although they don't necessarily require a callback if you don't need to know when the operation was performed.
var levelup = require('levelup')
var leveldown = require('leveldown')
var db = levelup(leveldown('./mydb'))
db.put('name', 'LevelUP', function (err) {
if (err) return console.log('Ooops!', err)
db.get('name', function (err, value) {
if (err) return console.log('Ooops!', err)
console.log('name=' + value)
})
})
API
Special Notes
levelup(db[, options[, callback]])
levelup()
is the main entry point for creating a new LevelUP instance and opening the underlying store with LevelDB.
db
is an abstract-leveldown
compliant object.
This function returns a new instance of LevelUP and will also initiate an open()
operation. Opening the database is an asynchronous operation which will trigger your callback if you provide one. The callback should take the form: function (err, db) {}
where the db
is the LevelUP instance. If you don't provide a callback, any read & write operations are simply queued internally until the database is fully opened.
This leads to two alternative ways of managing a new LevelUP instance:
levelup(leveldown(location), options, function (err, db) {
if (err) throw err
db.get('foo', function (err, value) {
if (err) return console.log('foo does not exist')
console.log('got foo =', value)
})
})
var db = levelup(leveldown(location), options)
db.get('foo', function (err, value) {
if (err) return console.log('foo does not exist')
console.log('got foo =', value)
})
options
is passed on to the underlying store when it's opened.
db.open([callback])
open()
opens the underlying store. In general you should never need to call this method directly as it's automatically called by levelup()
.
However, it is possible to reopen a database after it has been closed with close()
, although this is not generally advised.
If no callback is passed, a promise is returned.
db.close([callback])
close()
closes the underlying store. The callback will receive any error encountered during closing as the first argument.
You should always clean up your LevelUP instance by calling close()
when you no longer need it to free up resources. A LevelDB store cannot be opened by multiple instances of LevelDB/LevelUP simultaneously.
If no callback is passed, a promise is returned.
db.put(key, value[, options][, callback])
put()
is the primary method for inserting data into the store. Both the key
and value
can be arbitrary data objects.
options
is passed on to the underlying store.
If no callback is passed, a promise is returned.
db.get(key[, options][, callback])
get()
is the primary method for fetching data from the store. The key
can be an arbitrary data object. If it doesn't exist in the store then the callback or promise will receive an error. A not-found err object will be of type 'NotFoundError'
so you can err.type == 'NotFoundError'
or you can perform a truthy test on the property err.notFound
.
db.get('foo', function (err, value) {
if (err) {
if (err.notFound) {
return
}
return callback(err)
}
})
options
is passed on to the underlying store.
If no callback is passed, a promise is returned.
db.del(key[, options][, callback])
del()
is the primary method for removing data from the store.
db.del('foo', function (err) {
if (err)
});
options
is passed on to the underlying store.
If no callback is passed, a promise is returned.
db.batch(array[, options][, callback]) (array form)
batch()
can be used for very fast bulk-write operations (both put and delete). The array
argument should contain a list of operations to be executed sequentially, although as a whole they are performed as an atomic operation inside LevelDB.
Each operation is contained in an object having the following properties: type
, key
, value
, where the type is either 'put'
or 'del'
. In the case of 'del'
the 'value'
property is ignored. Any entries with a 'key'
of null
or undefined
will cause an error to be returned on the callback
and any 'type': 'put'
entry with a 'value'
of null
or undefined
will return an error.
If key
and value
are defined but type
is not, it will default to 'put'
.
var ops = [
{ type: 'del', key: 'father' },
{ type: 'put', key: 'name', value: 'Yuri Irsenovich Kim' },
{ type: 'put', key: 'dob', value: '16 February 1941' },
{ type: 'put', key: 'spouse', value: 'Kim Young-sook' },
{ type: 'put', key: 'occupation', value: 'Clown' }
]
db.batch(ops, function (err) {
if (err) return console.log('Ooops!', err)
console.log('Great success dear leader!')
})
options
is passed on to the underlying store.
If no callback is passed, a promise is returned.
db.batch() (chained form)
batch()
, when called with no arguments will return a Batch
object which can be used to build, and eventually commit, an atomic LevelDB batch operation. Depending on how it's used, it is possible to obtain greater performance when using the chained form of batch()
over the array form.
db.batch()
.del('father')
.put('name', 'Yuri Irsenovich Kim')
.put('dob', '16 February 1941')
.put('spouse', 'Kim Young-sook')
.put('occupation', 'Clown')
.write(function () { console.log('Done!') })
batch.put(key, value)
Queue a put operation on the current batch, not committed until a write()
is called on the batch.
This method may throw
a WriteError
if there is a problem with your put (such as the value
being null
or undefined
).
batch.del(key)
Queue a del operation on the current batch, not committed until a write()
is called on the batch.
This method may throw
a WriteError
if there is a problem with your delete.
batch.clear()
Clear all queued operations on the current batch, any previous operations will be discarded.
batch.length
The number of queued operations on the current batch.
batch.write([callback])
Commit the queued operations for this batch. All operations not cleared will be written to the database atomically, that is, they will either all succeed or fail with no partial commits.
If no callback is passed, a promise is returned.
db.isOpen()
A LevelUP object can be in one of the following states:
- "new" - newly created, not opened or closed
- "opening" - waiting for the database to be opened
- "open" - successfully opened the database, available for use
- "closing" - waiting for the database to be closed
- "closed" - database has been successfully closed, should not be used
isOpen()
will return true
only when the state is "open".
db.isClosed()
See isOpen()
isClosed()
will return true
only when the state is "closing" or "closed", it can be useful for determining if read and write operations are permissible.
db.createReadStream([options])
Returns a Readable Stream of key-value pairs. A pair is an object with 'key'
and 'value'
properties. By default it will stream all entries in the database from start to end. Use the options described below to control the range, direction and results.
db.createReadStream()
.on('data', function (data) {
console.log(data.key, '=', data.value)
})
.on('error', function (err) {
console.log('Oh my!', err)
})
.on('close', function () {
console.log('Stream closed')
})
.on('end', function () {
console.log('Stream ended')
})
You can supply an options object as the first parameter to createReadStream()
with the following properties:
-
'gt'
(greater than), 'gte'
(greater than or equal) define the lower bound of the range to be streamed. Only entries where the key is greater than (or equal to) this option will be included in the range. When reverse=true
the order will be reversed, but the entries streamed will be the same.
-
'lt'
(less than), 'lte'
(less than or equal) define the higher bound of the range to be streamed. Only entries where the key is less than (or equal to) this option will be included in the range. When reverse=true
the order will be reversed, but the entries streamed will be the same.
-
'reverse'
(boolean, default: false
): stream entries in reverse order. Beware that due to the way LevelDB works, a reverse seek will be slower than a forward seek.
-
'limit'
(number, default: -1
): limit the number of entries collected by this stream. This number represents a maximum number of entries and may not be reached if you get to the end of the range first. A value of -1
means there is no limit. When reverse=true
the entries with the highest keys will be returned instead of the lowest keys.
-
'keys'
(boolean, default: true
): whether the results should contain keys. If set to true
and 'values'
set to false
then results will simply be keys, rather than objects with a 'key'
property. Used internally by the createKeyStream()
method.
-
'values'
(boolean, default: true
): whether the results should contain values. If set to true
and 'keys'
set to false
then results will simply be values, rather than objects with a 'value'
property. Used internally by the createValueStream()
method.
Legacy options:
db.createKeyStream([options])
Returns a Readable Stream of keys rather than key-value pairs. Use the same options as described for createReadStream
to control the range and direction.
You can also obtain this stream by passing an options object to createReadStream()
with keys
set to true
and values
set to false
. The result is equivalent; both streams operate in object mode.
db.createKeyStream()
.on('data', function (data) {
console.log('key=', data)
})
db.createReadStream({ keys: true, values: false })
.on('data', function (data) {
console.log('key=', data)
})
db.createValueStream([options])
Returns a Readable Stream of values rather than key-value pairs. Use the same options as described for createReadStream
to control the range and direction.
You can also obtain this stream by passing an options object to createReadStream()
with values
set to true
and keys
set to false
. The result is equivalent; both streams operate in object mode.
db.createValueStream()
.on('data', function (data) {
console.log('value=', data)
})
db.createReadStream({ keys: false, values: true })
.on('data', function (data) {
console.log('value=', data)
})
What happened to db.createWriteStream
?
db.createWriteStream()
has been removed in order to provide a smaller and more maintainable core. It primarily existed to create symmetry with db.createReadStream()
but through much discussion, removing it was the best course of action.
The main driver for this was performance. While db.createReadStream()
performs well under most use cases, db.createWriteStream()
was highly dependent on the application keys and values. Thus we can't provide a standard implementation and encourage more write-stream
implementations to be created to solve the broad spectrum of use cases.
Check out the implementations that the community has already produced here.
Promise Support
LevelUp ships with native Promise
support out of the box.
Each function taking a callback also can be used as a promise, if the callback is omitted. This applies for:
db.get(key[, options])
db.put(key, value[, options])
db.del(key[, options])
db.batch(ops[, options])
db.batch().write()
The only exception is the levelup
constructor itself, which if no callback is passed will lazily open the database backend in the background.
Example:
var db = levelup(leveldown('./my-db'))
db.put('foo', 'bar')
.then(function () { return db.get('foo') })
.then(function (value) { console.log(value) })
.catch(function (err) { console.error(err) })
Or using async/await
:
const main = async () {
const db = levelup(leveldown('./my-db'))
await db.put('foo', 'bar')
console.log(await db.get('foo'))
}
Events
LevelUP emits events when the callbacks to the corresponding methods are called.
db.emit('put', key, value)
emitted when a new value is 'put'
db.emit('del', key)
emitted when a value is deleteddb.emit('batch', ary)
emitted when a batch operation has executeddb.emit('ready')
emitted when the database has opened ('open'
is synonym)db.emit('closed')
emitted when the database has closeddb.emit('opening')
emitted when the database is openingdb.emit('closing')
emitted when the database is closing
If you do not pass a callback to an async function, and there is an error, LevelUP will emit('error', err)
instead.
Extending LevelUP
A list of Node.js LevelDB modules and projects can be found in the wiki.
When attempting to extend the functionality of LevelUP, it is recommended that you consider using level-hooks and/or level-sublevel. level-sublevel is particularly helpful for keeping additional, extension-specific, data in a LevelDB store. It allows you to partition a LevelUP instance into multiple sub-instances that each correspond to discrete namespaced key ranges.
Multi-process access
LevelDB is thread-safe but is not suitable for accessing with multiple processes. You should only ever have a LevelDB database open from a single Node.js process. Node.js clusters are made up of multiple processes so a LevelUP instance cannot be shared between them either.
See the wiki for some LevelUP extensions, including multilevel, that may help if you require a single store to be shared across processes.
Getting support
There are multiple ways you can find help in using LevelDB in Node.js:
- IRC: you'll find an active group of LevelUP users in the ##leveldb channel on Freenode, including most of the contributors to this project.
- Mailing list: there is an active Node.js LevelDB Google Group.
- GitHub: you're welcome to open an issue here on this GitHub repository if you have a question.
Contributing
LevelUP is an OPEN Open Source Project. This means that:
Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
See the contribution guide for more details.
Windows
A large portion of the Windows support comes from code by Krzysztof Kowalczyk @kjk, see his Windows LevelDB port here. If you're using LevelUP on Windows, you should give him your thanks!
License & copyright
Copyright © 2012-2017 LevelUP contributors.
LevelUP is licensed under the MIT license. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE.md
file for more details.
LevelUP builds on the excellent work of the LevelDB and Snappy teams from Google and additional contributors. LevelDB and Snappy are both issued under the New BSD Licence.