NestDB
The Node.js Embedded JavaScript Database
Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, written in 100% JavaScript, no binary dependency. Originally forked from NeDB. API is similar to MongoDB's and it's plenty fast.
IMPORTANT NOTE: Please don't submit issues for questions regarding your code. Only actual bugs or feature requests will be answered, all others will be closed without comment. Also, please follow the bug reporting guidelines and check the change log before submitting an already fixed bug :)
Installation
Node.js version via NPM
$ npm install nestdb --save
Node.js version via Yarn
$ yarn add nestdb
Browser version via Bower
$ bower install nestdb --save
Compatibility with NeDB
NestDB was originally forked from NeDB.
NestDB v2.x
will maintain backward compatibility with NeDB v1.x
to make for seamless transitions migrating from NeDB to NestDB.
When we eventually release NestDB v3.x
, it will not be backward compatible with NeDB v1.x
.
API
It is similar to MongoDB's API (the most used operations). The NestDB API is a superset of NeDB's API.
- Creating/loading a datastore
- Persistence
- Inserting documents
- Finding documents
- Basic Querying
- Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex)
- Array fields
- Logical operators $or, $and, $not, $where
- Sorting and paginating
- Projections
- Counting documents
- Updating documents
- Removing documents
- Indexing
- Destroying a datastore
- Extending with plugins
- Using a custom storage engine
Creating/loading a datastore
You can use NestDB as an in-memory only datastore or as a persistent datastore. One datastore is the equivalent of a MongoDB collection. The constructor is used as follows new Datastore(options)
where options
is an object with the following fields:
filename
(optional): path to the file where the data is persisted. If left blank, the datastore is automatically considered in-memory only. It cannot end with a ~
which is used in the temporary files NestDB uses to perform crash-safe writes.inMemoryOnly
(optional, defaults to false
): as the name implies.timestampData
(optional, defaults to false
): timestamp the insertion and last update of all documents, with the fields createdAt
and updatedAt
. User-specified values override automatic generation, usually useful for testing.autoload
(optional, defaults to false
): if used, the datastore will automatically be loaded from the datafile upon creation (you don't need to call load
). Any command issued before load is finished is buffered and will be executed when load is done.onload
(optional): if you use autoloading, this is the handler called after the load
. It takes one error
argument. If you use autoloading without specifying this handler, and an error happens during load, an error will be thrown.idGenerator
(optional): if set, this function will be used for generating IDs. It takes no arguments and should return a unique string.afterSerialization
(optional): hook you can use to transform data after it was serialized and before it is written to disk. Can be used for example to encrypt data before writing datastore to disk. This function takes a string as parameter (one line of an NestDB data file) and outputs the transformed string, which must absolutely not contain a \n
character (or data will be lost).beforeDeserialization
(optional): inverse of afterSerialization
. Make sure to include both and not just one or you risk data loss. For the same reason, make sure both functions are inverses of one another. Some failsafe mechanisms are in place to prevent data loss if you misuse the serialization hooks: NestDB checks that never one is declared without the other, and checks that they are reverse of one another by testing on random strings of various lengths. In addition, if too much data is detected as corrupt, NestDB will refuse to start as it could mean you're not using the deserialization hook corresponding to the serialization hook used before (see below).corruptAlertThreshold
(optional): between 0 (0%) and 1 (100%), defaults to 0.1 (10%). NestDB will refuse to start if more than this percentage of the datafile is corrupt. 0 means you don't tolerate any corruption, 1 means you don't care.compareStrings
(optional): function compareStrings(a, b)
should compare strings a
and b
and must return -1
, 0
or 1
. If specified, it overrides default string comparison (===
), which is not well adapted to non-US characters such as accented or diacritical letters. Using the native String.prototype.localeCompare
will be the right choice most of the time.storage
(optional): A custom storage engine for the database files. Must implement at least the handful of methods exported by the standard "storage" module included in NestDB, as detailed below in the Using a custom storage engine section.
If you use a persistent datastore without the autoload
option, you need to call load
manually.
This function fetches the data from datafile and prepares the datastore. Do NOT forget it! If you use a
persistent datastore, no command (e.g. insert
, find
, update
, remove
) will be executed before load
is called, so make sure to either call it yourself or use the autoload
option.
Also, if load
fails, all commands registered to the executor afterwards will not be executed. They will be registered and executed, in sequence, only after a successful load
.
Once the datastore is fully loaded, it also emits a "loaded"
event that you can add a listener for.
Whenever any datastore is fully loaded, the Datastore
object itself emits a global "created"
event that you can add a listener for.
Type 1: In-memory only datastore (no need to load)
var Datastore = require('nestdb')
, db = new Datastore();
Type 2: Persistent datastore with manual loading
var Datastore = require('nestdb')
, db = new Datastore({ filename: 'path/to/datafile' });
db.load(function (err) {
});
Type 3: Persistent datastore with automatic loading
var Datastore = require('nestdb')
, db = new Datastore({
filename: 'path/to/datafile'
, autoload: true
, onload: function (err) {
if (err) {
console.error('Failed to load the datastore:', err);
} else {
console.log('Loaded the datastore!');
}
}
});
db.once('loaded', function () {
console.log('Loaded the datastore!');
});
Type 4: Persistent datastore for a Node WebKit app
var Datastore = require('nestdb')
, path = require('path')
, db = new Datastore({ filename: path.join(require('nw.gui').App.dataPath, 'something.db') });
Loading a datastore with an event listener
var Datastore = require('nestdb')
, db = new Datastore({ filename: 'path/to/datafile', autoload: true });
db.once('loaded', function () {
console.log('Loaded the datastore!');
});
Listening for the global "created" event
var Datastore = require('nestdb')
, db;
Datastore.on('created', function (dbRef) {
console.log('Created or loaded a datastore: ' + (dbRef.filename || 'in-memory'));
});
db = new Datastore({ filename: 'path/to/datafile', autoload: true });
Loading multiple datastores
db = {};
db.users = new Datastore('path/to/users.db');
db.robots = new Datastore('path/to/robots.db');
db.users.load();
db.robots.load();
Persistence
Under the hood, NestDB's persistence uses an append-only format, meaning that all updates and deletes actually result in lines added at the end of the datafile, for performance reasons. The datastore is automatically compacted (i.e. put back in the one-line-per-document format) every time you load each datastore within your application.
You can manually call the compaction function with yourDatastore.persistence.compactDatafile(cb)
, where callback is optional and get passed an error if any. It queues a compaction of the datafile in the executor, to be executed sequentially after all pending operations. The datastore will also fire a "compacted"
event whenever a compaction process is finished.
You can also set automatic compaction at regular intervals with yourDatastore.persistence.setAutocompactionInterval(interval)
, interval
in milliseconds (a minimum of 5 seconds is enforced), and stop automatic compaction with yourDatastore.persistence.stopAutocompaction()
.
Keep in mind that compaction takes a bit of time (not too much: 130ms for 50k records on a typical development machine) and no other operation can happen when it does, so most projects actually don't need to use it.
Compaction will also immediately remove any documents whose data line has become corrupted, assuming that the total percentage of all corrupted documents in that datastore still falls below the specified corruptAlertThreshold
option's value.
Durability works similarly to major databases: compaction forces the OS to physically flush data to disk, while appends to the data file do not (the OS is responsible for flushing the data). That guarantees that a server crash can never cause complete data loss, while preserving performance. The worst that can happen is a crash between two syncs, causing a loss of all data between the two syncs. Usually syncs are 30 seconds appart so that's at most 30 seconds of data. This post by Antirez on Redis persistence explains this in more details, NestDB being very close to Redis AOF persistence with appendfsync
option set to no
.
Inserting documents
The supported native types are String
, Number
, Boolean
, Date
and null
. You can also use arrays and subdocuments (objects). If a field is undefined
, it will not be saved (this is different from MongoDB which transforms undefined
in null
, something I find counter-intuitive).
If the document does not contain an _id
field, NestDB will automatically generate one for you (a 16-character alphanumerical string). The _id
of a document, once set, cannot be modified.
Field names cannot begin by '$' nor contain a '.'.
Once a document is fully inserted, its datastore will also emit an "inserted"
event that you can add a listener for. This event is always emitted once for each single new document, even if a bulk-insert is done.
Example 1: Inserting a single document
var doc = { hello: 'world'
, n: 5
, today: new Date()
, nestdbIsAwesome: true
, notthere: null
, notToBeSaved: undefined
, fruits: [ 'apple', 'orange', 'pear' ]
, infos: { name: 'nestdb' }
};
db.insert(doc, function (err, newDoc) {
});
Example 2: Inserting multiple documents
You can also bulk-insert an array of documents. This operation is atomic, meaning that if one insert fails due to a unique constraint being violated, all changes are rolled back.
db.insert([{ a: 5 }, { a: 42 }], function (err, newDocs) {
});
db.insert([{ a: 8 }, { a: 14 }, { a: 8 }], function (err) {
});
Example 3: Inserting a document with an event listener
db.on('inserted', function (newDoc) {
});
db.insert({ a: 5 });
db.insert({ a: 42 });
db.insert([{ a: 8 }, { a: 14 }]);
Finding documents
Use find
to look for multiple documents matching your query, or findOne
to look for one specific document. You can select documents based on field equality or use comparison operators ($lt
, $lte
, $gt
, $gte
, $in
, $nin
, $ne
). You can also use logical operators $or
, $and
, $not
and $where
. See below for the syntax.
You can use regular expressions in two ways: in basic querying in place of a string, or with the $regex
operator.
You can sort and paginate results using the cursor API (see below).
You can use standard projections to restrict the fields to appear in the results (see below).
Basic querying
Basic querying means are looking for documents whose fields match the ones you specify. You can use regular expression to match strings.
You can use the dot notation to navigate inside nested documents, arrays, arrays of subdocuments and to match a specific element of an array.
db.find({ system: 'solar' }, function (err, docs) {
});
db.find({ planet: /ar/ }, function (err, docs) {
});
db.find({ system: 'solar', inhabited: true }, function (err, docs) {
});
db.find({ "humans.genders": 2 }, function (err, docs) {
});
db.find({ "completeData.planets.name": "Mars" }, function (err, docs) {
});
db.find({ "completeData.planets.name": "Jupiter" }, function (err, docs) {
});
db.find({ "completeData.planets.0.name": "Earth" }, function (err, docs) {
});
db.find({ humans: { genders: 2 } }, function (err, docs) {
});
db.find({}, function (err, docs) {
});
db.findOne({ _id: 'id1' }, function (err, doc) {
});
Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex)
The syntax is { field: { $op: value } }
where $op
is any comparison operator:
$lt
, $lte
: less than, less than or equal$gt
, $gte
: greater than, greater than or equal$in
: member of. value
must be an array of values$ne
, $nin
: not equal, not a member of$exists
: checks whether the document posses the property field
. value
should be true or false$regex
: checks whether a string is matched by the regular expression. Contrary to MongoDB, the use of $options
with $regex
is not supported, because it doesn't give you more power than regex flags. Basic queries are more readable so only use the $regex
operator when you need to use another operator with it (see example below)
db.find({ "humans.genders": { $gt: 5 } }, function (err, docs) {
});
db.find({ planet: { $gt: 'Mercury' }}, function (err, docs) {
})
db.find({ planet: { $in: ['Earth', 'Jupiter'] }}, function (err, docs) {
});
db.find({ satellites: { $exists: true } }, function (err, docs) {
});
db.find({ planet: { $regex: /ar/, $nin: ['Jupiter', 'Earth'] } }, function (err, docs) {
});
Array fields
When a field in a document is an array, NestDB first tries to see if the query value is an array to perform an exact match, then whether there is an array-specific comparison function (for now there is only $size
and $elemMatch
) being used. If not, the query is treated as a query on every element and there is a match if at least one element matches.
$size
: match on the size of the array$elemMatch
: matches if at least one array element matches the query entirely
db.find({ satellites: ['Phobos', 'Deimos'] }, function (err, docs) {
})
db.find({ satellites: ['Deimos', 'Phobos'] }, function (err, docs) {
})
db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: 3 } } } }, function (err, docs) {
});
db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: 5 } } } }, function (err, docs) {
});
db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: { $gt: 2 } } } } }, function (err, docs) {
});
db.find({ satellites: { $size: 2 } }, function (err, docs) {
});
db.find({ satellites: { $size: 1 } }, function (err, docs) {
});
db.find({ satellites: 'Phobos' }, function (err, docs) {
});
db.find({ satellites: { $lt: 'Amos' } }, function (err, docs) {
});
db.find({ satellites: { $in: ['Moon', 'Deimos'] } }, function (err, docs) {
});
Logical operators $or, $and, $not, $where
You can combine queries using logical operators:
- For
$or
and $and
, the syntax is { $op: [query1, query2, ...] }
. - For
$not
, the syntax is { $not: query }
- For
$where
, the syntax is { $where: function () { /* object is "this", return a boolean */ } }
db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }] }, function (err, docs) {
});
db.find({ $not: { planet: 'Earth' } }, function (err, docs) {
});
db.find({ $where: function () { return Object.keys(this).length > 6; } }, function (err, docs) {
});
db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }], inhabited: true }, function (err, docs) {
});
Sorting and paginating
If you don't specify a callback to find
, findOne
or count
, a Cursor
object is returned. You can modify the cursor with sort
, skip
and limit
and then execute it with exec(callback)
.
db.find({}).sort({ planet: 1 }).skip(1).limit(2).exec(function (err, docs) {
});
db.find({ system: 'solar' }).sort({ planet: -1 }).exec(function (err, docs) {
});
db.find({}).sort({ firstField: 1, secondField: -1 }) ...
Projections
You can give find
and findOne
an optional second argument, projections
. The syntax is the same as MongoDB: { a: 1, b: 1 }
to return only the a
and b
fields, { a: 0, b: 0 }
to omit these two fields. You cannot use both modes at the time, except for _id
which is by default always returned and which you can choose to omit. You can project on nested documents.
db.find({ planet: 'Mars' }, { planet: 1, system: 1 }, function (err, docs) {
});
db.find({ planet: 'Mars' }, { planet: 1, system: 1, _id: 0 }, function (err, docs) {
});
db.find({ planet: 'Mars' }, { planet: 0, system: 0, _id: 0 }, function (err, docs) {
});
db.find({ planet: 'Mars' }, { planet: 0, system: 1 }, function (err, docs) {
});
db.find({ planet: 'Mars' }).projection({ planet: 1, system: 1 }).exec(function (err, docs) {
});
db.findOne({ planet: 'Earth' }).projection({ planet: 1, 'humans.genders': 1 }).exec(function (err, doc) {
});
Counting documents
You can use count
to count documents. It has the same syntax as find
. For example:
db.count({ system: 'solar' }, function (err, count) {
});
db.count({}, function (err, count) {
});
Updating documents
db.update(query, update, options, callback)
will update all documents matching query
according to the update
rules:
query
is the same kind of finding query you use with find
and findOne
update
specifies how the documents should be modified. It is either a new document or a set of modifiers (you cannot use both together, it doesn't make sense!)
- A new document will replace the matched docs
- The modifiers create the fields they need to modify if they don't exist, and you can apply them to subdocs. Available field modifiers are
$set
to change a field's value, $unset
to delete a field, $inc
to increment a field's value and $min
/$max
to change field's value, only if provided value is less/greater than current value. To work on arrays, you have $push
, $pop
, $addToSet
, $pull
, and the special $each
and $slice
. See examples below for the syntax.
options
is an object with two possible parameters
multi
(defaults to false
) which allows the modification of several documents if set to trueupsert
(defaults to false
) if you want to insert a new document corresponding to the update
rules if your query
doesn't match anything. If your update
is a simple object with no modifiers, it is the inserted document. In the other case, the query
is stripped from all operator recursively, and the update
is applied to it.returnUpdatedDocs
(defaults to false
, not MongoDB-compatible) if set to true and update is not an upsert, will return the array of documents matched by the find query and updated. Updated documents will be returned even if the update did not actually modify them.
callback
(optional) signature: (err, numAffected, affectedDocuments, upsert)
. Warning: the API was changed between v1.7.4
and v1.8.0
. Please refer to the change log to see the change.
- For an upsert,
affectedDocuments
contains the inserted document and the upsert
flag is set to true
. - For a standard update with
returnUpdatedDocs
flag set to false
, affectedDocuments
is not set. - For a standard update with
returnUpdatedDocs
flag set to true
and multi
to false
, affectedDocuments
is the updated document. - For a standard update with
returnUpdatedDocs
flag set to true
and multi
to true
, affectedDocuments
is the array of updated documents.
Note: You cannot change a document's _id
.
Once a document is fully updated, its datastore will also emit an "updated"
event that you can add a listener for. This event is always emitted once for each single updated document, even if a bulk-update is done.
If a document is upserted rather than updated, the datastore will emit an "inserted"
event instead.
Examples
db.update({ planet: 'Jupiter' }, { planet: 'Pluton'}, {}, function (err, numReplaced) {
});
db.update({ system: 'solar' }, { $set: { system: 'solar system' } }, { multi: true }, function (err, numReplaced) {
});
db.update({ planet: 'Mars' }, { $set: { "data.satellites": 2, "data.red": true } }, {}, function () {
db.update({ planet: 'Mars' }, { $set: { data: { satellites: 3 } } }, {}, function () {
});
});
db.update({ planet: 'Mars' }, { $unset: { planet: true } }, {}, function () {
});
db.update({ planet: 'Pluton' }, { planet: 'Pluton', inhabited: false }, { upsert: true }, function (err, numReplaced, upsert) {
});
db.update({ planet: 'Pluton' }, { $inc: { distance: 38 } }, { upsert: true }, function () {
});
db.update({ _id: 'id6' }, { $push: { fruits: 'banana' } }, {}, function () {
});
db.update({ _id: 'id6' }, { $pop: { fruits: 1 } }, {}, function () {
});
db.update({ _id: 'id6' }, { $addToSet: { fruits: 'apple' } }, {}, function () {
});
db.update({ _id: 'id6' }, { $pull: { fruits: 'apple' } }, {}, function () {
});
db.update({ _id: 'id6' }, { $pull: { fruits: $in: ['apple', 'pear'] } }, {}, function () {
});
db.update({ _id: 'id6' }, { $push: { fruits: { $each: ['banana', 'orange'] } } }, {}, function () {
});
db.update({ _id: 'id6' }, { $push: { fruits: { $each: ['banana'], $slice: 2 } } }, {}, function () {
});
db.update({ _id: 'id1' }, { $min: { value: 2 } }, {}, function () {
});
db.update({ _id: 'id1' }, { $min: { value: 8 } }, {}, function () {
});
db.on('updated', function (newDoc, oldDoc) {
});
db.insert({ a: 5 });
db.update({ a: 5 }, { $set: { a: 8 } });
Removing documents
db.remove(query, options, callback)
will remove all documents matching query
according to options
query
is the same as the ones used for finding and updatingoptions
only one option for now: multi
which allows the removal of multiple documents if set to true. Default is falsecallback
is optional, signature: err, numRemoved
Once a document is fully removed, its datastore will also emit an "removed"
event that you can add a listener for. This event is always emitted once for each single removed document, even if a bulk-remove is done.
db.remove({ _id: 'id2' }, {}, function (err, numRemoved) {
});
db.remove({ system: 'solar' }, { multi: true }, function (err, numRemoved) {
});
db.remove({}, { multi: true }, function (err, numRemoved) {
});
db.on('removed', function (oldDoc) {
});
db.insert({ a: 5 });
db.insert({ a: 8 });
db.insert({ a: 14 });
db.remove({ $min: { a: 8 } }, { multi: true });
Indexing
NestDB supports indexing. It gives a very nice speed boost and can be used to enforce a unique constraint on a field. You can index any field, including fields in nested documents using the dot notation. For now, indexes are only used to speed up basic queries and queries using $in
, $lt
, $lte
, $gt
and $gte
. The indexed values cannot be of type array of object.
To create an index, use datastore.ensureIndex(options, cb)
, where callback is optional and get passed an error if any (usually a unique constraint that was violated). ensureIndex
can be called when you want, even after some data was inserted, though it's best to call it at application startup. The options are:
- fieldName (required): name of the field to index. Use the dot notation to index a field in a nested document.
- unique (optional, defaults to
false
): enforce field uniqueness. Note that a unique index will raise an error if you try to index two documents for which the field is not defined. - sparse (optional, defaults to
false
): don't index documents for which the field is not defined. Use this option along with "unique" if you want to accept multiple documents for which it is not defined. - expireAfterSeconds (number of seconds, optional): if set, the created index is a TTL (time to live) index, that will automatically remove documents when the system date becomes larger than the date on the indexed field plus
expireAfterSeconds
. Documents where the indexed field is not specified or not a Date
object are ignored
Note: the _id
is automatically indexed with a unique constraint, no need to call ensureIndex
on it.
You can remove a previously created index with datastore.removeIndex(fieldName, cb)
.
If your datastore is persistent, the indexes you created are persisted in the datafile, when you load the datastore a second time they are automatically created for you. No need to remove any ensureIndex
though, if it is called on a datastore that already has the index, nothing happens.
db.ensureIndex({ fieldName: 'somefield' }, function (err) {
});
db.ensureIndex({ fieldName: 'somefield', unique: true }, function (err) {
});
db.ensureIndex({ fieldName: 'somefield', unique: true, sparse: true }, function (err) {
});
db.insert({ somefield: 'nestdb' }, function (err) {
db.insert({ somefield: 'nestdb' }, function (err) {
});
});
db.removeIndex('somefield', function (err) {
});
db.ensureIndex({ fieldName: 'createdAt', expireAfterSeconds: 3600 }, function (err) {
});
db.ensureIndex({ fieldName: 'expirationDate', expireAfterSeconds: 0 }, function (err) {
});
Note: the ensureIndex
function creates the index synchronously, so it's best to use it at application startup. It's quite fast so it doesn't increase startup time much (35 ms for a collection containing 10,000 documents).
Destroying a datastore
If you ever decide you want to destroy an existing datastore, you can call destroy
.
Destroying a datastore will:
- Remove all documents
- Reset all indexes
- Delete the datafile, if persistent
Once the datastore is fully destroyed, it also emits a "destroyed"
event that you can add a listener for.
Whenever any datastore is fully destroyed, the Datastore
object itself emits a global "destroyed"
event that you can add a listener for.
Type 1: In-memory only datastore
var Datastore = require('nestdb')
, db = new Datastore();
db.destroy();
Type 2: Persistent datastore destroyed with a callback
var Datastore = require('nestdb')
, db = new Datastore({ filename: 'path/to/datafile', autoload: true });
db.destroy(function (err) {
if (err) {
console.error('Failed to destroy datastore:', err);
} else {
console.log('Destroyed the datastore!');
}
});
Type 3: Persistent datastore destroyed with an event listener
var Datastore = require('nestdb')
, db = new Datastore({ filename: 'path/to/datafile', autoload: true });
db.once('destroyed', function () {
console.log('Destroyed the datastore!');
});
db.destroy();
Listening for the global "destroyed" event
var Datastore = require('nestdb')
, db = new Datastore({ filename: 'path/to/datafile', autoload: true });
Datastore.on('destroyed', function (dbRef) {
console.log('Destroyed a datastore: ' + (dbRef.filename || 'in-memory'));
});
db.destroy();
Extending with plugins
Extending NestDB with plugins is easy! There are two different implementation patterns available for plugins.
Regardless of which pattern you choose, the Datastore.plugin()
method returns the Datastore
object itself, making it easy to chain multiple plugin calls serially.
API extension object
First, you can pass in an object. Each key in the object will be added as a property to the Datastore.prototype
. It intentionally does NOT check for conflicts, so be mindful when naming your properties that you may be overwriting an existing property.
Datastore.plugin({
sayHello: function () {
console.log('Hello!');
}
});
var db = new Datastore();
db.sayHello();
This will add a sayHello
instance method to all datastores, which runs associated function. It will always be called in context, so that within the function, this
refers to the datastore instance.
API extension function
Alternatively, instead of passing in an object to Datastore.plugin()
, you can pass in a function that takes the Datastore object and performs whatever operations you want on it. You can use this to load multiple plugins, add adapters, or attach event listeners to the Datastore object.
Datastore.plugin(function (Datastore) {
Datastore.hello = function () {
console.log('world');
};
Datastore.prototype.goodbye = function () {
console.log('cruel world');
};
});
Datastore.hello();
var db = new Datastore();
db.goodbye();
Using a custom storage engine
As mentioned in the Creating/loading a datastore section above, NestDB allows you to specify a custom storage engine for a persistent Datastore
instance by passing in a custom options.storage
object to the constructor.
This is useful for leveraging NestDB as a database wrapper around non-standard file systems, such as React Native or WinJS.
The required interface to implement for a custom storage engine is as follows:
storage.init
Purpose: Immediately before loading a datastore, ensure that either:
- a datafile exists
- The default implementation also ensures that the datafile contains all the data even if there was a crash during a full file write by attempting to recover it from a temporary datafile if needed.
- or a new empty datafile is created
- The default implementation also takes advantage of this to create any missing directories in the file's path.
Interface
This function must accept the following exact invocation: storage.init(file, callback)
file
: Required. A String representing the path to a datafile.callback
: Required. A Function that MUST be invoked when the operation has completed.
- Its first invocation argument, if any, must either be an
Error
instance or undefined
/null
.
The function's return value is irrelevant.
storage.read
Purpose: Read all contents from a datafile during the initial loading process for a datastore. The datafile should separate its records using "\n"
as the delimiting character.
Interface
This function must accept the following exact invocation: storage.read(file, callback)
file
: Required. A String representing the path to a datafile.callback
: Required. A Function that MUST be invoked when the operation has completed.
- Its first invocation argument must either be an
Error
instance or undefined
/null
. - Its second invocation argument must be a String representing the entire datafile's contents, which will be turned into records by splitting on the
"\n"
delimiting characters.
The function's return value is irrelevant.
storage.append
Purpose: Append new records to a datafile without modifying its current contents. This handles all insertions, modifications, and deletions (by appending "action" records) for both documents and index definitions.
Interface
This function must accept the following exact invocation: storage.append(file, data, callback)
file
: Required. A String representing the path to a datafile.data
: Required. A String representing the new contents to be added to the datafile. Multiple records must be provided with "\n"
delimiting characters.callback
: Required. A Function that MUST be invoked when the operation has completed.
- Its first invocation argument, if any, must either be an
Error
instance or undefined
/null
.
The function's return value is irrelevant.
storage.write
Purpose: Use to overwrite the entire contents of the datafile after processing and condensing all records. This is used at the end of the initial loading process, as well as during every compaction operation.
Interface
This function must accept the following exact invocation: storage.write(file, data, callback)
file
: Required. A String representing the path to a datafile.data
: Required. A String representing the entire contents to be written to the datafile. Multiple records must be provided with "\n"
delimiting characters.callback
: Required. A Function that MUST be invoked when the operation has completed.
- Its first invocation argument, if any, must either be an
Error
instance or undefined
/null
.
The function's return value is irrelevant.
storage.remove
Purpose: Completely delete a datafile, if it exists. This is only used if a datastore is intentionally destroyed.
Interface
This function must accept the following exact invocation: storage.remove(file, callback)
file
: Required. A String representing the path to a datafile.callback
: Required. A Function that MUST be invoked when the operation has completed.
- Its first invocation argument, if any, must either be an
Error
instance or undefined
/null
.
The function's return value is irrelevant.
Browser version
The browser version and its minified counterpart are in the browser-version/out/
directory. You only need to require nestdb.js
or nestdb.min.js
in your HTML file and the global object NestDB
can be used right away, with the same API as the server version:
<script src="nestdb.min.js"></script>
<script>
var db = new NestDB();
db.insert({ planet: 'Earth' }, function (err) {
db.find({}, function (err, docs) {
});
});
</script>
If you specify a filename
, the datastore will be persistent, and automatically select the best storage method available (IndexedDB, WebSQL or localStorage) depending on the browser. In most cases that means a lot of data can be stored, typically in hundreds of MB. WARNING: the storage system changed between v1.3 and v1.4 and is NOT back-compatible! Your application needs to resync client-side when you upgrade NestDB.
NestDB is compatible with all major browsers: Chrome, Safari, Firefox, IE9+. Tests are in the browser-version/test
directory (files index.html
and testPersistence.html
).
If you fork and modify nestdb, you can build the browser version from the sources, the build script is npm run build
.
Performance
Speed
NestDB is not intended to be a replacement of large-scale databases such as MongoDB, and as such was not designed for speed. That said, it is still pretty fast on the expected data sets, especially if you use indexing. On a typical, not-so-fast dev machine, for a collection containing 10,000 documents, with indexing:
- Insert: 10,680 ops/s
- Find: 43,290 ops/s
- Update: 8,000 ops/s
- Remove: 11,750 ops/s
You can run these simple benchmarks by executing the scripts in the benchmarks
folder. Run them with the --help
flag to see how they work.
A copy of the whole datastore is kept in memory. This is not much on the
expected kind of data sets (20MB for 10,000 2KB documents).
NeDB use in other services
- connect-nedb-session is a session store for Connect and Express, backed by nedb
- If you mostly use NeDB for logging purposes and don't want the memory footprint of your application to grow too large, you can use NeDB Logger to insert documents in a NeDB-readable datastore
- If you've outgrown NeDB, switching to MongoDB won't be too hard as it is the same API. Use this utility to transfer the data from a NeDB datastore to a MongoDB collection
- An ODM for NeDB: Camo
Pull requests
If you submit a pull request, thanks! There are a couple rules to follow though to make it manageable:
- The pull request should be atomic, i.e. contain only one feature. If it contains more, please submit multiple pull requests. Reviewing massive, 1000 loc+ pull requests is extremely hard.
- Likewise, if for one unique feature the pull request grows too large (more than 200 loc tests not included), please get in touch first.
- Please stick to the current coding style. It's important that the code uses a coherent style for readability.
- Do not include stylistic improvements ("housekeeping"). If you think one part deserves lots of housekeeping, use a separate pull request so as not to pollute the code.
- Don't forget tests for your new feature. Also don't forget to run the whole test suite before submitting to make sure you didn't introduce regressions.
- Do not build the browser version in your branch, I'll take care of it once the code is merged.
- Update the readme accordingly.
- Last but not least: keep in mind what NestDB's mindset is! The goal is not to be a replacement for MongoDB, but to have a pure JS database, easy to use, cross platform, fast and expressive enough for the target projects (small and self contained apps on server/desktop/browser/mobile). Sometimes it's better to shoot for simplicity than for API completeness with regards to MongoDB.
Bug reporting guidelines
If you report a bug, thank you! That said for the process to be manageable please strictly adhere to the following guidelines. I'll not be able to handle bug reports that don't:
- Your bug report should be a self-containing gist complete with a package.json for any dependencies you need. I need to run through a simple
git clone gist; npm install; node bugreport.js
, nothing more. - It should use assertions to showcase the expected vs actual behavior and be hysteresis-proof. It's quite simple in fact, see this example: https://gist.github.com/louischatriot/220cf6bd29c7de06a486
- Simplify as much as you can. Strip all your application-specific code. Most of the time you will see that there is no bug but an error in your code :)
- 50 lines max. If you need more, read the above point and rework your bug report. If you're really convinced you need more, please explain precisely in the issue.
- The code should be Javascript, not Coffeescript.
License
See License