Embedded JSON Database engine
It aims to be a fast MongoDB-like library which can be embedded into C/C++ and NodeJS applications under terms of LGPL license.
EJDB is the C library based on modified version of Tokyo Cabinet.
JSON representation of queries and data implemented with API based on C BSON
Features
- LGPL license allows you to embed this library into proprietary software.
- MongoDB-like queries and overall philosophy.
- Collection level write locking.
- Collection level transactions.
- String token matching queries:
$stror
$strand
- Node.js binding
Documentation
We use EJDB Google group as our mailing list.
NodeJS binding
One snippet intro
var EJDB = require("ejdb");
var jb = EJDB.open("zoo", EJDB.DEFAULT_OPEN_MODE | EJDB.JBOTRUNC);
var parrot1 = {
"name" : "Grenny",
"type" : "African Grey",
"male" : true,
"age" : 1,
"birthdate" : new Date(),
"likes" : ["green color", "night", "toys"],
"extra1" : null
};
var parrot2 = {
"name" : "Bounty",
"type" : "Cockatoo",
"male" : false,
"age" : 15,
"birthdate" : new Date(),
"likes" : ["sugar cane"]
};
jb.save("parrots", [parrot1, parrot2], function(err, oids) {
if (err) {
console.error(err);
return;
}
console.log("Grenny OID: " + parrot1["_id"]);
console.log("Bounty OID: " + parrot2["_id"]);
jb.find("parrots",
{"likes" : "toys"},
{"$orderby" : {"name" : 1}},
function(err, cursor, count) {
if (err) {
console.error(err);
return;
}
console.log("Found " + count + " parrots");
while (cursor.next()) {
console.log(cursor.field("name") + " likes toys!");
}
cursor.close();
jb.close();
});
});
Installation
System libraries:
On Debian/Ubuntu linux you can install it as follows:
sudo apt-get install g++ zlib1g zlib1g-dev
Installation from node package manager:
npm install ejdb
EJDB NodeJS samples
EJDB NodeJS API
### EJDB.open(dbFile, openMode)
Open database. Return database instance handle object.
Default open mode: JBOWRITER | JBOCREAT
.
This is blocking function.
Arguments
- {String} dbFile Database main file name
- {Number}
[openMode=JBOWRITER | JBOCREAT]
Bitmast of open modes:
- JBOREADER
Open as a reader.
- JBOWRITER
Open as a writer.
- JBOCREAT
Create if db file not exists
- JBOTRUNC
Truncate db.
### close()
Close database.
If database was not opened it does nothing.
This is blocking function.
### isOpen()
Check if database in opened state.
### ensureCollection(cname, copts)
Automatically creates new collection if it does't exists.
Collection options copts
applied only for newly created collection.
For existing collections copts
takes no effect.
Collection options (copts):
- cachedrecords : Max number of cached records in shared memory segment. Default: 0
- records : Estimated number of records in this collection. Default: 65535.
- large : Specifies that the size of the database can be larger than 2GB. Default: false
- compressed : If true collection records will be compressed with DEFLATE compression. Default: false.
This is blocking function.
Arguments
- {String} cname Name of collection.
- {Object}
[copts]
Collection options.
### dropCollection(cname, prune, cb)
Drop collection.
Call variations:
dropCollection(cname)
dropCollection(cname, cb)
dropCollection(cname, prune, cb)
Arguments
- {String} cname Name of collection.
- {Boolean}
[prune=false]
If true the collection data will erased from disk. - {Function}
[cb]
Callback args: (error)
### save(cname, jsarr, cb)
Save/update specified JSON objects in the collection.
If collection with cname
does not exists it will be created.
Each persistent object has unique identifier (OID) placed in the _id
property.
If a saved object does not have _id
it will be autogenerated.
To identify and update object it should contains _id
property.
If callback is not provided this function will be synchronous.
Call variations:
save(cname, json object, [cb])
save(cname, <Array of json objects>, [cb])
Arguments
- {String} cname Name of collection.
- {Array|Object} jsarr Signle JSON object or array of JSON objects to save
- {Function}
[cb]
Callback args: (error, {Array} of OIDs for saved objects)
Return
- {Array} of OIDs of saved objects in synchronous mode otherwise returns {undefined}.
### load(cname, oid, cb)
Loads JSON object identified by OID from the collection.
If callback is not provided this function will be synchronous.
Arguments
- {String} cname Name of collection
- {String} oid Object identifier (OID)
- {Function} cb Callback args: (error, obj)
obj
: Retrieved JSON object or NULL if it is not found.
Return
- JSON object or {null} if it is not found in synchronous mode otherwise return {undefined}.
### remove(cname, oid, cb)
Removes JSON object from the collection.
If callback is not provided this function will be synchronous.
Arguments
- {String} cname Name of collection
- {String} oid Object identifier (OID)
- {Function} cb Callback args: (error)
### find(cname, qobj, orarr, hints, cb)
Execute query on collection.
EJDB queries inspired by MongoDB (mongodb.org) and follows same philosophy.
Supported queries:
- Simple matching of String OR Number OR Array value:
- {'json.field.path' : 'val', ...}
- $not Negate operation.
- {'json.field.path' : {'$not' : val}} //Field not equal to val
- {'json.field.path' : {'$not' : {'$begin' : prefix}}} //Field not begins with val
- $begin String starts with prefix
- {'json.field.path' : {'$begin' : prefix}}
- $gt, $gte (>, >=) and $lt, $lte for number types:
- {'json.field.path' : {'$gt' : number}, ...}
- $bt Between for number types:
- {'json.field.path' : {'$bt' : [num1, num2]}}
- $in String OR Number OR Array val matches to value in specified array:
- {'json.field.path' : {'$in' : [val1, val2, val3]}}
- $nin - Not IN
- $strand String tokens OR String array val matches all tokens in specified array:
- {'json.field.path' : {'$strand' : [val1, val2, val3]}}
- $stror String tokens OR String array val matches any token in specified array:
- {'json.field.path' : {'$stror' : [val1, val2, val3]}}
- $exists Field existence matching:
- {'json.field.path' : {'$exists' : true|false}}
- $icase Case insensitive string matching:
- {'json.field.path' : {'$icase' : 'val1'}} //icase matching
Ignore case matching with '$in' operation:
- {'name' : {'$icase' : {'$in' : ['tHéâtre - театр', 'heLLo WorlD']}}}
For case insensitive matching you can create special type of string index.
- $elemMatch The $elemMatch operator matches more than one component within an array element.
- { array: { $elemMatch: { value1 : 1, value2 : { $gt: 1 } } } }
Restriction: only one $elemMatch allowed in context of one array field.
- Queries can be used to update records:
$set Field set operation.
- {.., '$set' : {'field1' : val1, 'fieldN' : valN}}
$upsert Atomic upsert. If matching records are found it will be '$set' operation,
otherwise new record will be inserted with fields specified by argment object.
- {.., '$upsert' : {'field1' : val1, 'fieldN' : valN}}
$inc Increment operation. Only number types are supported.
- {.., '$inc' : {'field1' : number, ..., 'field1' : number}
$dropall In-place record removal operation.
- {.., '$dropall' : true}
$addToSet Atomically adds value to the array only if its not in the array already.
If containing array is missing it will be created.
- {.., '$addToSet' : {'json.field.path' : val1, 'json.field.pathN' : valN, ...}}
$pull - Atomically removes all occurrences of value from field, if field is an array.
- {.., '$pull' : {'json.field.path' : val1, 'json.field.pathN' : valN, ...}}
NOTE: It is better to execute update queries with `$onlycount=true` hint flag
or use the special `update()` method to avoid unnecessarily rows fetching.
NOTE: Negate operations: $not and $nin not using indexes
so they can be slow in comparison to other matching operations.
NOTE: Only one index can be used in search query operation.
NOTE: If callback is not provided this function will be synchronous.
QUERY HINTS (specified by `hints` argument):
- $max Maximum number in the result set
- $skip Number of skipped results in the result set
- $orderby Sorting order of query fields.
- $onlycount true|false If `true` only count of matching records will be returned
without placing records in result set.
- $fields Set subset of fetched fields
Example:
hints: {
"$orderby" : { //ORDER BY field1 ASC, field2 DESC
"field1" : 1,
"field2" : -1
},
"$fields" : { //SELECT ONLY {_id, field1, field2}
"field1" : 1,
"field2" : 1
}
}
Many C API query examples can be found in `tcejdb/testejdb/t2.c` test case.
To traverse selected records cursor object is used:
- Cursor#next() Move cursor to the next record and returns true if next record exists.
- Cursor#hasNext() Returns true if cursor can be placed to the next record.
- Cursor#field(name) Retrieve value of the specified field of the current JSON object record.
- Cursor#object() Retrieve whole JSON object with all fields.
- Cursor#reset() Reset cursor to its initial state.
- Cursor#length Read-only property: Number of records placed into cursor.
- Cursor#pos Read/Write property: You can set cursor position: 0 <= pos < length
- Cursor#close() Closes cursor and free cursor resources. Cursor cant be used in closed state.
Call variations of find():
- find(cname, [cb])
- find(cname, qobj, [cb])
- find(cname, qobj, hints, [cb])
- find(cname, qobj, qobjarr, [cb])
- find(cname, qobj, qobjarr, hints, [cb])
Arguments
- {String} cname Name of collection
- {Object} qobj Main JSON query object
- {Array}
[orarr]
Array of additional OR query objects (joined with OR predicate). - {Object}
[hints]
JSON object with query hints. - {Function} cb Callback args: (error, cursor, count)
cursor
: Cursor object to traverse records
qobj count
: Total number of selected records
Return
- If callback is provided returns {undefined}
- If no callback and
$onlycount
hint is set returns count {Number}. - If no callback and no
$onlycount
hint returns cursor {Object}.
### findOne(cname, qobj, orarr, hints, cb)
Same as #find() but retrieves only one matching JSON object.
Call variations of findOne():
findOne(cname, [cb])
findOne(cname, qobj, [cb])
findOne(cname, qobj, hints, [cb])
findOne(cname, qobj, qobjarr, [cb])
findOne(cname, qobj, qobjarr, hints, [cb])
Arguments
- {String} cname Name of collection
- {Object} qobj Main JSON query object
- {Array}
[orarr]
Array of additional OR query objects (joined with OR predicate). - {Object}
[hints]
JSON object with query hints. - {Function} cb Callback args: (error, obj)
obj
Retrieved JSON object or NULL if it is not found.
Return
- If callback is provided returns {undefined}
- If no callback is provided returns found {Object} or {null}
### update(cname, qobj, orarr, hints, cb)
Convenient method to execute update queries.
$set
Field set operation:
- {some fields for selection, '$set' : {'field1' : {obj}, ..., 'field1' : {obj}}}
$upsert
Atomic upsert. If matching records are found it will be '$set' operation,
otherwise new record will be inserted with fields specified by argment object.
- {.., '$upsert' : {'field1' : val1, 'fieldN' : valN}}
$inc
Increment operation. Only number types are supported.
- {some fields for selection, '$inc' : {'field1' : number, ..., 'field1' : {number}}
$dropall
In-place record removal operation.
- {some fields for selection, '$dropall' : true}
$addToSet
Atomically adds value to the array only if its not in the array already.
If containing array is missing it will be created.
- {.., '$addToSet' : {'json.field.path' : val1, 'json.field.pathN' : valN, ...}}
$pull
Atomically removes all occurrences of value from field, if field is an array.
- {.., '$pull' : {'json.field.path' : val1, 'json.field.pathN' : valN, ...}}
Call variations of update():
update(cname, [cb])
update(cname, qobj, [cb])
update(cname, qobj, hints, [cb])
update(cname, qobj, qobjarr, [cb])
update(cname, qobj, qobjarr, hints, [cb])
Arguments
- {String} cname Name of collection
- {Object} qobj Update JSON query object
- {Array}
[orarr]
Array of additional OR query objects (joined with OR predicate). - {Object}
[hints]
JSON object with query hints. - {Function} cb Callback args: (error, count)
count
The number of updated records.
Return
- If callback is provided returns {undefined}.
- If no callback is provided returns {Number} of updated objects.
### count(cname, qobj, orarr, hints, cb)
Convenient count(*) operation.
Call variations of count():
count(cname, [cb])
count(cname, qobj, [cb])
count(cname, qobj, hints, [cb])
count(cname, qobj, qobjarr, [cb])
count(cname, qobj, qobjarr, hints, [cb])
Arguments
- {String} cname Name of collection
- {Object} qobj Main JSON query object
- {Array}
[orarr]
Array of additional OR query objects (joined with OR predicate). - {Object}
[hints]
JSON object with query hints. - {Function} cb Callback args: (error, count)
count
: Number of matching records.
Return
- If callback is provided returns {undefined}.
- If no callback is provided returns {Number} of matched object.
### sync(cb)
Synchronize entire EJDB database with disk.
Arguments
- {Function} cb Callback args: (error)
### dropIndexes(cname, path, cb)
Drop indexes of all types for JSON field path.
Arguments
- {String} cname Name of collection
- {String} path JSON field path
- {Function}
[cb]
Optional callback function. Callback args: (error)
### optimizeIndexes(cname, path, cb)
Optimize indexes of all types for JSON field path.
Performs B+ tree index file optimization.
Arguments
- {String} cname Name of collection
- {String} path JSON field path
- {Function}
[cb]
Optional callback function. Callback args: (error)
### ensureStringIndex(cname, path, cb)
### ensureIStringIndex(cname, path, cb)
### ensureNumberIndex(cname, path, cb)
### ensureArrayIndex(cname, path, cb)
Ensure index presence of String|Number|Array type for JSON field path.
IString
is the special type of String index for case insensitive matching.
Arguments
- {String} cname Name of collection
- {String} path JSON field path
- {Function}
[cb]
Optional callback function. Callback args: (error)
### rebuildStringIndex(cname, path, cb)
### rebuildIStringIndex(cname, path, cb)
### rebuildNumberIndex(cname, path, cb)
### rebuildArrayIndex(cname, path, cb)
Rebuild index of String|Number|Array type for JSON field path.
IString
is the special type of String index for case insensitive matching.
Arguments
- {String} cname Name of collection
- {String} path JSON field path
- {Function}
[cb]
Optional callback function. Callback args: (error)
### dropStringIndex(cname, path, cb)
### dropIStringIndex(cname, path, cb)
### dropNumberIndex(cname, path, cb)
### dropArrayIndex(cname, path, cb)
Drop index of String|Number|Array type for JSON field path.
IString
is the special type of String index for case insensitive matching.
Arguments
- {String} cname Name of collection
- {String} path JSON field path
- {Function}
[cb]
Optional callback function. Callback args: (error)
EJDB C Library
One snippet intro
#include <tcejdb/ejdb.h>
static EJDB *jb;
int main() {
jb = ejdbnew();
if (!ejdbopen(jb, "addressbook", JBOWRITER | JBOCREAT | JBOTRUNC)) {
return 1;
}
//Get or create collection 'contacts'
EJCOLL *coll = ejdbcreatecoll(jb, "contacts", NULL);
bson bsrec;
bson_oid_t oid;
//Insert one record:
//JSON: {'name' : 'Bruce', 'phone' : '333-222-333', 'age' : 58}
bson_init(&bsrec);
bson_append_string(&bsrec, "name", "Bruce");
bson_append_string(&bsrec, "phone", "333-222-333");
bson_append_int(&bsrec, "age", 58);
bson_finish(&bsrec);
//Save BSON
ejdbsavebson(coll, &bsrec, &oid);
fprintf(stderr, "\nSaved Bruce");
bson_destroy(&bsrec);
//Now execute query
//QUERY: {'name' : {'$begin' : 'Bru'}} //Name starts with 'Bru' string
bson bq1;
bson_init_as_query(&bq1);
bson_append_start_object(&bq1, "name");
bson_append_string(&bq1, "$begin", "Bru");
bson_append_finish_object(&bq1);
bson_finish(&bq1);
EJQ *q1 = ejdbcreatequery(jb, &bq1, NULL, 0, NULL);
uint32_t count;
TCLIST *res = ejdbqrysearch(coll, q1, &count, 0, NULL);
fprintf(stderr, "\n\nRecords found: %d\n", count);
//Now print the result set records
for (int i = 0; i < TCLISTNUM(res); ++i) {
void *bsdata = TCLISTVALPTR(res, i);
bson_print_raw(stderr, bsdata, 0);
}
fprintf(stderr, "\n");
//Dispose result set
tclistdel(res);
//Dispose query
ejdbquerydel(q1);
bson_destroy(&bq1);
//Close database
ejdbclose(jb);
ejdbdel(jb);
return 0;
}
You can save this code in csnippet.c
And build:
gcc -std=c99 -Wall -pedantic -c -o csnippet.o csnippet.c
gcc -std=c99 -Wall -pedantic -o csnippet csnippet.o -ltcejdb
Building & Installation
Prerequisites
System libraries:
On Debian/Ubuntu linux you can install it as follows:
sudo apt-get install gcc zlib1g zlib1g-dev
Building
cd ./tcejdb
./configure --prefix=<installation prefix> && make && make check
make install
- library name: tcejdb (with pkgconfig)
- main include header:
<tcejdb/ejdb.h>
C API
EJDB API presented in ejdb.h C header file.
JSON processing API: bson.h
Queries
/**
* Create query object.
* Sucessfully created queries must be destroyed with ejdbquerydel().
*
* EJDB queries inspired by MongoDB (mongodb.org) and follows same philosophy.
*
* - Supported queries:
* - Simple matching of String OR Number OR Array value:
* - {'json.field.path' : 'val', ...}
* - $not Negate operation.
* - {'json.field.path' : {'$not' : val}} //Field not equal to val
* - {'json.field.path' : {'$not' : {'$begin' : prefix}}} //Field not begins with val
* - $begin String starts with prefix
* - {'json.field.path' : {'$begin' : prefix}}
* - $gt, $gte (>, >=) and $lt, $lte for number types:
* - {'json.field.path' : {'$gt' : number}, ...}
* - $bt Between for number types:
* - {'json.field.path' : {'$bt' : [num1, num2]}}
* - $in String OR Number OR Array val matches to value in specified array:
* - {'json.field.path' : {'$in' : [val1, val2, val3]}}
* - $nin - Not IN
* - $strand String tokens OR String array val matches all tokens in specified array:
* - {'json.field.path' : {'$strand' : [val1, val2, val3]}}
* - $stror String tokens OR String array val matches any token in specified array:
* - {'json.field.path' : {'$stror' : [val1, val2, val3]}}
* - $exists Field existence matching:
* - {'json.field.path' : {'$exists' : true|false}}
* - $icase Case insensitive string matching:
* - {'json.field.path' : {'$icase' : 'val1'}} //icase matching
* Ignore case matching with '$in' operation:
* - {'name' : {'$icase' : {'$in' : ['tHéâtre - театр', 'heLLo WorlD']}}}
* For case insensitive matching you can create special index of type: `JBIDXISTR`
* - $elemMatch The $elemMatch operator matches more than one component within an array element.
* - { array: { $elemMatch: { value1 : 1, value2 : { $gt: 1 } } } }
* Restriction: only one $elemMatch allowed in context of one array field.
*
* - Queries can be used to update records:
* $set Field set operation.
* - {.., '$set' : {'field1' : val1, 'fieldN' : valN}}
* $upsert Atomic upsert. If matching records are found it will be '$set' operation,
* otherwise new record will be inserted with fields specified by argment object.
* - {.., '$upsert' : {'field1' : val1, 'fieldN' : valN}}
* $inc Increment operation. Only number types are supported.
* - {.., '$inc' : {'field1' : number, ..., 'field1' : number}
* $dropall In-place record removal operation.
* - {.., '$dropall' : true}
* $addToSet Atomically adds value to the array only if its not in the array already.
* If containing array is missing it will be created.
* - {.., '$addToSet' : {'json.field.path' : val1, 'json.field.pathN' : valN, ...}}
* $pull - Atomically removes all occurrences of value from field, if field is an array.
* - {.., '$pull' : {'json.field.path' : val1, 'json.field.pathN' : valN, ...}}
*
* NOTE: Negate operations: $not and $nin not using indexes
* so they can be slow in comparison to other matching operations.
*
* NOTE: Only one index can be used in search query operation.
*
* QUERY HINTS (specified by `hints` argument):
* - $max Maximum number in the result set
* - $skip Number of skipped results in the result set
* - $orderby Sorting order of query fields.
* - $fields Set subset of fetched fields
* Example:
* hints: {
* "$orderby" : { //ORDER BY field1 ASC, field2 DESC
* "field1" : 1,
* "field2" : -1
* },
* "$fields" : { //SELECT ONLY {_id, field1, field2}
* "field1" : 1,
* "field2" : 1
* }
* }
*
* Many query examples can be found in `testejdb/t2.c` test case.
*
* @param EJDB database handle.
* @param qobj Main BSON query object.
* @param orqobjs Array of additional OR query objects (joined with OR predicate).
* @param orqobjsnum Number of OR query objects.
* @param hints BSON object with query hints.
* @return On success return query handle. On error returns NULL.
*/
EJDB_EXPORT EJQ* ejdbcreatequery(EJDB *jb, bson *qobj, bson *orqobjs, int orqobjsnum, bson *hints);
EJDB C Samples
You can find some code samples in:
Basic EJDB architecture
EJDB database files structure
.
├── <dbname>
├── <dbname>_<collection1>
├── ...
├── <dbname>_<collectionN>
└── <dbname>_<collectionN>_<fieldpath>.<index ext>
Where
<dbname>
- name of database. It is metadata DB.<collectionN>
- name of collection. Collection database.<fieldpath>
- JSON field path used in index<index ext>
- Collection index extension:
.lex
String index.dec
Number index.tok
Array index
Limitations/TODOs
- Collect collection index statistic
- Windows port
Related software
Connect session store backed by EJDB database