NeDB (Node embedded database)
Embedded persistent database for Node.js, written in Javascript, with no dependency (except npm
modules), which
can be used with a simple require
statement. The API is a subset of MongoDB's. You can use it as a persistent or an in-memory only datastore, and it can also be used in all recent browsers (Chrome, Firefox, Safari, IE9+).
NeDB is not intended to be a replacement of large-scale databases such as MongoDB! Its goal is to provide you with a clean and easy way to query data and persist it to disk, for web applications that do not need lots of concurrent connections, for example a continuous integration and deployment server and desktop applications built with Node Webkit.
NeDB was benchmarked against the popular client-side database TaffyDB and NeDB is much, much faster. That's why there is now a browser version, which can also provide persistence.
Check the change log in the wiki if you think nedb doesn't behave as the documentation describes! Most of the issues I get are due to non-latest version NeDBs.
Support NeDB development
No time to help out? You can support NeDB development by sending money or bitcoins!
Money:
Bitcoin address: 1dDZLnWpBbodPiN8sizzYrgaz5iahFyb1
Installation, tests
Module name on npm is nedb
.
npm install nedb --save
npm test
API
It's a subset of MongoDB's API (the most used operations). The current API will not change, but I will add operations as they are needed. Summary of the API:
- Creating/loading a database
- Compacting the database
- Inserting documents
- Finding documents
- Basic Querying
- Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex)
- Array fields
- Logical operators $or, $and, $not, $where
- Sorting and paginating
- Projections
- Counting documents
- Updating documents
- Removing documents
- Indexing
- Browser version
Creating/loading a database
You can use NeDB as an in-memory only datastore or as a persistent datastore. One datastore is the equivalent of a MongoDB collection. The constructor is used as follows new Datastore(options)
where options
is an object with the following fields:
filename
(optional): path to the file where the data is persisted. If left blank, the datastore is automatically considered in-memory only. It cannot end with a ~
which is used in the temporary files NeDB uses to perform crash-safe writesinMemoryOnly
(optional, defaults to false): as the name implies.autoload
(optional, defaults to false): if used, the database will
automatically be loaded from the datafile upon creation (you don't
need to call loadDatabase
). Any command
issued before load is finished is buffered and will be executed when
load is done.onload
(optional): if you use autoloading, this is the handler called after the loadDatabase
. It takes one error
argument. If you use autoloading without specifying this handler, and an error happens during load, an error will be thrown.afterSerialization
(optional): hook you can use to transform data after it was serialized and before it is written to disk. Can be used for example to encrypt data before writing database to disk. This function takes a string as parameter (one line of an NeDB data file) and outputs the transformed string, which must absolutely not contain a \n
character (or data will be lost)beforeDeserialization
(optional): reverse of afterSerialization
. Make sure to include both and not just one or you risk data loss. For the same reason, make sure both functions are inverses of one another. Some failsafe mechanisms are in place to prevent data loss if you misuse the serialization hooks: NeDB checks that never one is declared without the other, and checks that they are reverse of one another by testing on random strings of various lengths. In addition, if too much data is detected as corrupt, NeDB will refuse to start as it could mean you're not using the deserialization hook corresponding to the serialization hook used before (see below)corruptAlertThreshold
(optional): between 0 and 1, defaults to 10%. NeDB will refuse to start if more than this percentage of the datafile is corrupt. 0 means you don't tolerate any corruption, 1 means you don't carenodeWebkitAppName
(optional, DEPRECATED): if you are using NeDB from whithin a Node Webkit app, specify its name (the same one you use in the package.json
) in this field and the filename
will be relative to the directory Node Webkit uses to store the rest of the application's data (local storage etc.). It works on Linux, OS X and Windows. Now that you can use require('nw.gui').App.dataPath
in Node Webkit to get the path to the data directory for your application, you should not use this option anymore and it will be removed.
If you use a persistent datastore without the autoload
option, you need to call loadDatabase
manually.
This function fetches the data from datafile and prepares the database. Don't forget it! If you use a
persistent datastore, no command (insert, find, update, remove) will be executed before loadDatabase
is called, so make sure to call it yourself or use the autoload
option.
var Datastore = require('nedb')
, db = new Datastore();
var Datastore = require('nedb')
, db = new Datastore({ filename: 'path/to/datafile' });
db.loadDatabase(function (err) {
});
var Datastore = require('nedb')
, db = new Datastore({ filename: 'path/to/datafile', autoload: true });
var Datastore = require('nedb')
, path = require('path')
, db = new Datastore({ filename: path.join(require('nw.gui').App.dataPath, 'something.db') });
db = {};
db.users = new Datastore('path/to/users.db');
db.robots = new Datastore('path/to/robots.db');
db.users.loadDatabase();
db.robots.loadDatabase();
Compacting the database
Under the hood, NeDB's persistence uses an append-only format, meaning that all updates and deletes actually result in lines added at the end of the datafile. The reason for this is that disk space is very cheap and appends are much faster than rewrites since they don't do a seek. The database is automatically compacted (i.e. put back in the one-line-per-document format) everytime your application restarts.
You can manually call the compaction function with yourDatabase.persistence.compactDatafile
which takes no argument. It queues a compaction of the datafile in the executor, to be executed sequentially after all pending operations.
You can also set automatic compaction at regular intervals with yourDatabase.persistence.setAutocompactionInterval(interval)
, interval
in milliseconds (a minimum of 5s is enforced), and stop automatic compaction with yourDatabase.persistence.stopAutocompaction()
.
Keep in mind that compaction takes a bit of time (not too much: 130ms for 50k records on my slow machine) and no other operation can happen when it does, so most projects actually don't need to use it.
Inserting documents
The native types are String
, Number
, Boolean
, Date
and null
. You can also use
arrays and subdocuments (objects). If a field is undefined
, it will not be saved (this is different from
MongoDB which transforms undefined
in null
, something I find counter-intuitive).
If the document does not contain an _id
field, NeDB will automatically generated one for you (a 16-characters alphanumerical string). The _id
of a document, once set, cannot be modified.
Field names cannot begin by '$' or contain a '.'.
var doc = { hello: 'world'
, n: 5
, today: new Date()
, nedbIsAwesome: true
, notthere: null
, notToBeSaved: undefined
, fruits: [ 'apple', 'orange', 'pear' ]
, infos: { name: 'nedb' }
};
db.insert(doc, function (err, newDoc) {
});
You can also bulk-insert an array of documents. This operation is atomic, meaning that if one insert fails due to a unique constraint being violated, all changes are rolled back.
db.insert([{ a: 5 }, { a: 42 }], function (err, newDocs) {
});
db.insert([{ a: 5 }, { a: 42 }, { a: 5 }], function (err) {
});
Finding documents
Use find
to look for multiple documents matching you query, or findOne
to look for one specific document. You can select documents based on field equality or use comparison operators ($lt
, $lte
, $gt
, $gte
, $in
, $nin
, $ne
). You can also use logical operators $or
, $and
, $not
and $where
. See below for the syntax.
You can use regular expressions in two ways: in basic querying in place of a string, or with the $regex
operator.
You can sort and paginate results using the cursor API (see below).
You can use standard projections to restrict the fields to appear in the results (see below).
Basic querying
Basic querying means are looking for documents whose fields match the ones you specify. You can use regular expression to match strings.
You can use the dot notation to navigate inside nested documents, arrays, arrays of subdocuments and to match a specific element of an array.
db.find({ system: 'solar' }, function (err, docs) {
});
db.find({ planet: /ar/ }, function (err, docs) {
});
db.find({ system: 'solar', inhabited: true }, function (err, docs) {
});
db.find({ "humans.genders": 2 }, function (err, docs) {
});
db.find({ "completeData.planets.name": "Mars" }, function (err, docs) {
});
db.find({ "completeData.planets.name": "Jupiter" }, function (err, docs) {
});
db.find({ "completeData.planets.0.name": "Earth" }, function (err, docs) {
});
db.find({ humans: { genders: 2 } }, function (err, docs) {
});
db.find({}, function (err, docs) {
});
db.findOne({ _id: 'id1' }, function (err, doc) {
});
Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex)
The syntax is { field: { $op: value } }
where $op
is any comparison operator:
$lt
, $lte
: less than, less than or equal$gt
, $gte
: greater than, greater than or equal$in
: member of. value
must be an array of values$ne
, $nin
: not equal, not a member of$exists
: checks whether the document posses the property field
. value
should be true or false$regex
: checks whether a string is matched by the regular expression. Contrary to MongoDB, the use of $options
with $regex
is not supported, because it doesn't give you more power than regex flags. Basic queries are more readable so only use the $regex
operator when you need to use another operator with it (see example below)
db.find({ "humans.genders": { $gt: 5 } }, function (err, docs) {
});
db.find({ planet: { $gt: 'Mercury' }}, function (err, docs) {
})
db.find({ planet: { $in: ['Earth', 'Jupiter'] }}, function (err, docs) {
});
db.find({ satellites: { $exists: true } }, function (err, docs) {
});
db.find({ planet: { $regex: /ar/, $nin: ['Jupiter', 'Earth'] } }, function (err, docs) {
});
Array fields
When a field in a document is an array, NeDB first tries to see if there is an array-specific comparison function (for now there is only $size
) being used
and tries it first. If there isn't, the query is treated as a query on every element and there is a match if at least one element matches.
db.find({ satellites: { $size: 2 } }, function (err, docs) {
});
db.find({ satellites: { $size: 1 } }, function (err, docs) {
});
db.find({ satellites: 'Phobos' }, function (err, docs) {
});
db.find({ satellites: { $lt: 'Amos' } }, function (err, docs) {
});
db.find({ satellites: { $in: ['Moon', 'Deimos'] } }, function (err, docs) {
});
Logical operators $or, $and, $not, $where
You can combine queries using logical operators:
- For
$or
and $and
, the syntax is { $op: [query1, query2, ...] }
. - For
$not
, the syntax is { $not: query }
- For
$where
, the syntax is { $where: function () { /* object is "this", return a boolean */ } }
db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }] }, function (err, docs) {
});
db.find({ $not: { planet: 'Earth' } }, function (err, docs) {
});
db.find({ $where: function () { return Object.keys(this) > 6; } }, function (err, docs) {
});
db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }], inhabited: true }, function (err, docs) {
});
Sorting and paginating
If you don't specify a callback to find
, findOne
or count
, a Cursor
object is returned. You can modify the cursor with sort
, skip
and limit
and then execute it with exec(callback)
.
db.find({}).sort({ planet: 1 }).skip(1).limit(2).exec(function (err, docs) {
});
db.find({ system: 'solar' }).sort({ planet: -1 }).exec(function (err, docs) {
});
db.find({}).sort({ firstField: 1, secondField: -1 }) ...
Projections
You can give find
and findOne
an optional second argument, projections
. The syntax is the same as MongoDB: { a: 1, b: 1 }
to return only the a
and b
fields, { a: 0, b: 0 }
to omit these two fields. You cannot use both modes at the time, except for _id
which is by default always returned and which you can choose to omit.
db.find({ planet: 'Mars' }, { planet: 1, system: 1 }, function (err, docs) {
});
db.find({ planet: 'Mars' }, { planet: 1, system: 1, _id: 0 }, function (err, docs) {
});
db.find({ planet: 'Mars' }, { planet: 0, system: 0, _id: 0 }, function (err, docs) {
});
db.find({ planet: 'Mars' }, { planet: 0, system: 1 }, function (err, docs) {
});
db.find({ planet: 'Mars' }).projection({ planet: 1, system: 1 }).exec(function (err, docs) {
});
Counting documents
You can use count
to count documents. It has the same syntax as find
. For example:
db.count({ system: 'solar' }, function (err, count) {
});
db.count({}, function (err, count) {
});
Updating documents
db.update(query, update, options, callback)
will update all documents matching query
according to the update
rules:
query
is the same kind of finding query you use with find
and findOne
update
specifies how the documents should be modified. It is either a new document or a set of modifiers (you cannot use both together, it doesn't make sense!)
- A new document will replace the matched docs
- The modifiers create the fields they need to modify if they don't exist, and you can apply them to subdocs. Available field modifiers are
$set
to change a field's value, $unset
to delete a field and $inc
to increment a field's value. To work on arrays, you have $push
, $pop
, $addToSet
, $pull
, and the special $each
. See examples below for the syntax.
options
is an object with two possible parameters
multi
(defaults to false
) which allows the modification of several documents if set to trueupsert
(defaults to false
) if you want to insert a new document corresponding to the update
rules if your query
doesn't match anything. If your update
is a simple object with no modifiers, it is the inserted document. In the other case, the query
is stripped from all operator recursively, and the update
is applied to it.
callback
(optional) signature: err
, numReplaced
, newDoc
numReplaced
is the number of documents replacednewDoc
is the created document if the upsert mode was chosen and a document was inserted
Note: you can't change a document's _id.
db.update({ planet: 'Jupiter' }, { planet: 'Pluton'}, {}, function (err, numReplaced) {
});
db.update({ system: 'solar' }, { $set: { system: 'solar system' } }, { multi: true }, function (err, numReplaced) {
});
db.update({ planet: 'Mars' }, { $set: { "data.satellites": 2, "data.red": true } }, {}, function () {
db.update({ planet: 'Mars' }, { $set: { data: { satellites: 3 } } }, {}, function () {
});
});
db.update({ planet: 'Mars' }, { $unset: { planet: true } }, {}, function () {
});
db.update({ planet: 'Pluton' }, { planet: 'Pluton', inhabited: false }, { upsert: true }, function (err, numReplaced, upsert) {
});
db.update({ planet: 'Pluton' }, { $inc: { distance: 38 } }, { upsert: true }, function () {
});
db.update({ _id: 'id6' }, { $push: { fruits: 'banana' } }, {}, function () {
});
db.update({ _id: 'id6' }, { $pop: { fruits: 1 } }, {}, function () {
});
db.update({ _id: 'id6' }, { $addToSet: { fruits: 'apple' } }, {}, function () {
});
db.update({ _id: 'id6' }, { $pull: { fruits: 'apple' } }, {}, function () {
});
db.update({ _id: 'id6' }, { $pull: { fruits: $in: ['apple', 'pear'] } }, {}, function () {
});
db.update({ _id: 'id6' }, { $push: { fruits: {$each: ['banana', 'orange'] } } }, {}, function () {
});
Removing documents
db.remove(query, options, callback)
will remove all documents matching query
according to options
query
is the same as the ones used for finding and updatingoptions
only one option for now: multi
which allows the removal of multiple documents if set to true. Default is falsecallback
is optional, signature: err, numRemoved
db.remove({ _id: 'id2' }, {}, function (err, numRemoved) {
});
db.remove({ system: 'solar' }, { multi: true }, function (err, numRemoved) {
});
Indexing
NeDB supports indexing. It gives a very nice speed boost and can be used to enforce a unique constraint on a field. You can index any field, including fields in nested documents using the dot notation. For now, indexes are only used to speed up basic queries and queries using $in
, $lt
, $lte
, $gt
and $gte
.
To create an index, use datastore.ensureIndex(options, cb)
, where callback is optional and get passed an error if any (usually a unique constraint that was violated). ensureIndex
can be called when you want, even after some data was inserted, though it's best to call it at application startup. The options are:
- fieldName (required): name of the field to index. Use the dot notation to index a field in a nested document.
- unique (optional, defaults to
false
): enforce field uniqueness. Note that a unique index will raise an error if you try to index two documents for which the field is not defined. - sparse (optional, defaults to
false
): don't index documents for which the field is not defined. Use this option along with "unique" if you want to accept multiple documents for which it is not defined.
Note: the _id
is automatically indexed with a unique constraint, no need to call ensureIndex
on it.
You can remove a previously created index with datastore.removeIndex(fieldName, cb)
.
If your datastore is persistent, the indexes you created are persisted in the datafile, when you load the database a second time they are automatically created for you. No need to remove any ensureIndex
though, if it is called on a database that already has the index, nothing happens.
db.ensureIndex({ fieldName: 'somefield' }, function (err) {
});
db.ensureIndex({ fieldName: 'somefield', unique: true }, function (err) {
});
db.ensureIndex({ fieldName: 'somefield', unique: true, sparse: true }, function (err) {
});
db.insert({ somefield: 'nedb' }, function (err) {
db.insert({ somefield: 'nedb' }, function (err) {
});
});
db.removeIndex('somefield', function (err) {
});
Note: the ensureIndex
function creates the index synchronously, so it's best to use it at application startup. It's quite fast so it doesn't increase startup time much (35 ms for a collection containing 10,000 documents).
Browser version
As of v0.8.0, you can use NeDB in the browser! You can find it and its minified version in the repository, in the browser-version/out
directory. You only need to require nedb.js
or nedb.min.js
in your HTML file and the global object Nedb
can be used right away, with the same API as the server version:
<script src="nedb.min.js"></script>
<script>
var db = new Nedb(); // Create an in-memory only datastore
db.insert({ planet: 'Earth' });
db.insert({ planet: 'Mars' });
db.find({}, function (err, docs) {
// docs contains the two planets Earth and Mars
});
</script>
It has been tested and is compatible with Chrome, Safari, Firefox, IE 10, IE 9. Please open an issue if you need compatibility with IE 8/IE 7, I think it will need some work and am not sure it is needed, since most complex webapplications - the ones that would need NeDB - only work on modern browsers anyway. To launch the tests, simply open the file browser-version/test/index.html
in a browser and you'll see the results of the tests for this browser.
If you fork and modify nedb, you can build the browser version from the sources, the build script is browser-version/build.js
.
As of v0.11, NeDB is also persistent on the browser. To use this, simply create the collection with the filename
option which will be the name of the localStorage
variable storing data. Persistence should work on all browsers where NeDB works. Also, keep in mind that localStorage
has size constraints, so it's probably a good idea to set recurring compaction every 2-5 minutes to save on space if your client app needs a lot of updates and deletes. See database compaction for more details on the append-only format used by NeDB.
Browser persistence is still young! It has been tested on most major browsers but please report any bugs you find
Performance
Speed
NeDB is not intended to be a replacement of large-scale databases such as MongoDB, and as such was not designed for speed. That said, it is still pretty fast on the expected datasets, especially if you use indexing. On my machine (3 years old, no SSD), with a collection containing 10,000 documents, with indexing:
- Insert: 5,950 ops/s
- Find: 25,440 ops/s
- Update: 4,490 ops/s
- Remove: 6,620 ops/s
You can run the simple benchmarks I use by executing the scripts in the benchmarks
folder. Run them with the --help
flag to see how they work.
A copy of the whole database is kept in memory. This is not much on the
expected kind of datasets (20MB for 10,000 2KB documents). If requested, I'll introduce an
option to not use this cache to decrease memory footprint (at the cost
of a lower speed).
Use in other services
- connect-nedb-session is a session store for
Connect and Express, backed by nedb
- If you mostly use NeDB for logging purposes and don't want the memory footprint of your application to grow too large, you can use NeDB Logger to insert documents in a NeDB-readable database
- If you've outgrown NeDB, switching to MongoDB won't be too hard as it is the same API. Use this utility to transfer the data from a NeDB database to a MongoDB collection
- An ODM for NeDB: Camo
Help out
Issues reporting and pull requests are always appreciated. For issues, make sure to always include a code snippet and describe the expected vs actual behavior. If you send a pull request, make sure to stick to NeDB's coding style and always test all the code you submit. You can look at the current tests to see how to do it
Bitcoins
You don't have time? You can support NeDB by sending bitcoins to this address: 1dDZLnWpBbodPiN8sizzYrgaz5iahFyb1
License
See License