mg-dbx-bdb
Advanced tools
High speed Synchronous and Asynchronous access to Berkeley DB and LMDB from Node.js.
High speed Synchronous and Asynchronous access to Berkeley DB and LMDB from Node.js.
Chris Munt cmunt@mgateway.com
21 May 2024, MGateway Ltd http://www.mgateway.com
Contents
There are two parts to this project. Firstly, simple synchronous and asynchronous access to Berkeley DB and/or LMDB B-Tree storage using either integer or string based keys. Secondly, an emulation of the M Global Storage Database is layered on top of these two B-Tree storage solutions. This Document describes in detail the Global Storage model and provides the rationale for the mg-dbx-bdb project.
In this mode, a Berkeley DB (or LMDB) database is created/opened and key-value pairs are added to it.
database[{keys}]=value
Examples:
db = person_database.db
db[1]="John Smith"
db[2]="Jane Jones"
The M Global Storage database is broadly B-Tree based but includes extra structural elements. The database is divided up into discrete sections known, in M, as global variables or "globals" for short. You can think of each M global as being conceptually similar to a table in a relational database. Records in each global are identified by a key. In the M model, each key can consist of one or many individual key items, giving the dataset the multidimensional characteristics of the M storage model. In M, keys are known as subscripts and are conceptually similar to a (primary) key in a relational database.
database[{global variables}, {keys ...}]=value
Examples:
db = database.db
person = new db.global("person")
hospital_admission = new db.global("admission")
person[1]="John Smith"
person[2]="Jane Jones"
hospital_admission[1, 20201203]="Admission Record for 3 December 2020"
hospital_admission[1, 20210101]="Admission Record for 1 January 2021"
The above scheme shows a one-to-many relationship between a person and that person's hospital admission records. All records are held in a single Berkeley DB (or LMDB) database.
mg-dbx-bdb is a Node.js addon written in C++. It is distributed as C++ source code and the NPM installation procedure will expect a C++ compiler to be present on the target system.
Linux systems can use the freely available GNU C++ compiler (g++) which can be installed as follows.
Ubuntu, Debian and Raspberry Pi OS:
apt-get install g++
Red Hat and CentOS:
yum install gcc-c++
Apple OS X can use the freely available Xcode development environment.
Microsoft Windows can use the freely available Microsoft Visual Studio Community:
If the Windows machine is not set up for systems development, building native Addon modules for this platform from C++ source can be quite arduous. There is some helpful advice available at:
Alternatively there are built Windows x64 binaries available from:
Assuming that Node.js is already installed and a C++ compiler is available to the installation process:
npm install mg-dbx-bdb
This command will create the mg-dbx-bdb addon (mg-dbx-bdb.node).
Most mg-dbx-bdb methods are capable of operating either synchronously or asynchronously. For an operation to complete asynchronously, simply supply a suitable callback as the last argument in the call.
The first step is to add mg-dbx-bdb to your Node.js script
var dbxbdb = require('mg-dbx-bdb').dbxbdb;
And optionally (as required):
var mglobal = require('mg-dbx-bdb').mglobal;
var mcursor = require('mg-dbx-bdb').mcursor;
var db = new dbxbdb();
To open a BDB database, the database file is specified in open() property db_file. This will allow single-process access to the database. In order to open a BDB database for concurrent access through multiple processes the BDB environment directory should be specified in open() property env_dir. In summary:
In the following examples, modify all paths to match those of your own installation.
Assuming BDB is installed under /usr/local/BerkeleyDB.18.1/lib/
var open = db.open({
type: "BDB",
db_library: "/usr/local/BerkeleyDB.18.1/lib/libdb.so",
db_file: "/opt/bdb/my_bdb_database.db",
env_dir: "/opt/bdb",
key_type: "int"});
});
Assuming BDB is installed under c:/bdb/
var open = db.open({
type: "BDB",
db_library: "c:/bdb/libdb181.dll",
db_file: "c:/bdb/my_bdb_database.db",
env_dir: "c:/bdb",
key_type: "int"});
});
To open a LMDB database for concurrent access, the database environment is specified in open() property env_dir. The database (and associated files) will be created in this directory.
In the following examples, modify all paths to match those of your own installation.
Assuming LMDB is installed in the usual location: /usr/lib/liblmdb.so
var open = db.open({
type: "LMDB",
db_library: "liblmdb.so",
env_dir: "/opt/lmdb",
key_type: "int"});
});
Assuming the LMDB DLL is installed as: c:/LMDBWindows/lib/LMDBWindowsDll64.dll
var open = db.open({
type: "LMDB",
db_library: "c:/LMDBWindows/lib/LMDBWindowsDll64.dll",
env_dir: "c:/lmdb",
key_type: "int"});
});
db_size: The maximum size for a LMDB environment/database. The default maximum database size for LMDB is 10M. However, for databases created using mg-dbx-bdb the default size will be set to 50M and this can be modified using this parameter.
multithreaded: A boolean value to be set to 'true' or 'false' (default multithreaded: true). Set this property to 'true' if the application uses multithreaded techniques in JavaScript (e.g. V8 worker threads).
dberror_exceptions: A boolean value to be set to 'true' or 'false' (default: dberror_exceptions: false). Set this property to 'true' to instruct mg-dbx-bdb to throw Node.js exceptions if synchronous invocation of database operations result in an error condition. If this property is not set, any error condition resulting from the previous database operation can be retrieved using the db.geterrormessage() method.
var result = db.version();
Example:
console.log("\nmg-dbx-bdb Version: " + db.version());
UTF-8 is the default character encoding for mg-dbx-bdb. The other option is the 8-bit ASCII character set (characters of the range ASCII 0 to ASCII 255). The ASCII character set is a better option when exchanging single-byte binary data with the database.
current_charset = db.charset([<new_charset>]);
Example 1 (Get the current character set):
var charset = db.charset();
Example 2 (Change the current character set):
var new_charset = db.charset('ascii');
db.close();
This mode of operation applies to BDB (or LMDB) databases opened with a key_type of int or str.
Synchronous:
var result = db.set(<key>, <data>);
Asynchronous:
db.set(<key>, <data>, callback(<error>, <result>));
Example:
db.set(1, "John Smith");
Synchronous:
var result = db.get(<key>);
Asynchronous:
db.get(<key>, callback(<error>, <result>));
Example:
var name = db.get(1);
Synchronous:
var result = db.delete(<key>);
Asynchronous:
db.delete(<key>, callback(<error>, <result>));
Example:
var name = db.delete(1);
Synchronous:
var result = db.defined(<key>);
Asynchronous:
db.defined(<key>, callback(<error>, <result>));
Example:
var name = db.defined(1);
Synchronous:
var result = db.next(<key>);
Asynchronous:
db.next(<key>, callback(<error>, <result>));
Example:
var key = "";
while ((key = db.next(key)) != "") {
console.log("\nPerson: " + key + ' : ' + db.get(key));
}
Synchronous:
var result = db.previous(<key>);
Asynchronous:
db.previous(<key>, callback(<error>, <result>));
Example:
var key = "";
while ((key = db.previous(key)) != "") {
console.log("\nPerson: " + key + ' : ' + db.get(key));
}
This facility provides high-performance techniques for traversing records held in the database.
The first task is to specify the 'query' for the database traverse.
query = new mcursor(db, {key: [<seed_key>]}[, <options>]);
Or:
query = db.mglobalquery({key: [<seed_key>]}[, <options>]);
The 'options' object can contain the following properties:
getdata: A boolean value (default: getdata: false). Set to 'true' to return data values associated with each database record returned.
format: Format for output (default: not specified). If the output consists of multiple data elements, the return value (by default) is a JavaScript object made up of a 'key' array and an associated 'data' value. Set to "url" to return such data as a single URL escaped string including the key value ('key1') and associated 'data' value.
Example (return all keys and names from the 'Person' global):
query = db.mglobalquery({key: [""]}, {multilevel: false, getdata: true});
In key order:
result = query.next();
In reverse key order:
result = query.previous();
In all cases these methods will return 'null' when the end of the dataset is reached.
Example 1 (return all key values from the database - returns a simple variable):
query = db.mglobalquery({key: [""]});
while ((result = query.next()) !== null) {
console.log("result: " + result);
}
Example 2 (return all key and data values from the database - returns an object):
query = db.mglobalquery({key: [""]}, getdata: true);
while ((result = query.next()) !== null) {
console.log("result: " + JSON.stringify(result, null, '\t'));
}
This mode of operation applies to BDB (or LMDB) databases opened with a key_type of m.
global = new mglobal(db, <global_name>[, <fixed_key>]);
Or:
global = db.mglobal(<global_name>[, <fixed_key>]);
Example (using a global named "Person"):
var person = db.mglobal("Person");
Synchronous:
var result = <global>.set(<key>, <data>);
Asynchronous:
<global>.set(<key>, <data>, callback(<error>, <result>));
Example:
person.set(1, "John Smith");
Synchronous:
var result = <global>.get(<key>);
Asynchronous:
<global>.get(<key>, callback(<error>, <result>));
Example:
var name = person.get(1);
Synchronous:
var result = <global>.delete(<key>);
Asynchronous:
<global>.delete(<key>, callback(<error>, <result>));
Example:
var name = person.delete(1);
Synchronous:
var result = <global>.defined(<key>);
Asynchronous:
<global>.defined(<key>, callback(<error>, <result>));
Example:
var name = person.defined(1);
Synchronous:
var result = <global>.next(<key>);
Asynchronous:
<global>.next(<key>, callback(<error>, <result>));
Example:
var key = "";
while ((key = person.next(key)) != "") {
console.log("\nPerson: " + key + ' : ' + person.get(key));
}
Synchronous:
var result = <global>.previous(<key>);
Asynchronous:
<global>.previous(<key>, callback(<error>, <result>));
Example:
var key = "";
while ((key = person.previous(key)) != "") {
console.log("\nPerson: " + key + ' : ' + person.get(key));
}
Synchronous:
var result = <global>.increment(<key>, <increment_value>);
Asynchronous:
<global>.increment(<key>, <increment_value>, callback(<error>, <result>));
Example (increment the value of the "counter" node by 1.5 and return the new value):
var result = person.increment("counter", 1.5);
Synchronous:
var result = <global>.lock(<key>, <timeout>);
Asynchronous:
<global>.lock(<key>, <timeout>, callback(<error>, <result>));
Example (lock global node '1' with a timeout of 30 seconds):
var result = person.lock(1, 30);
Synchronous:
var result = <global>.unlock(<key>);
Asynchronous:
<global>.unlock(<key>, callback(<error>, <result>));
Example (unlock global node '1'):
var result = person.unlock(1);
Synchronous (merge from global2 to global1):
var result = <global1>.merge([<key1>,] <global2> [, <key2>]);
Asynchronous (merge from global2 to global1):
<global1>.defined([<key1>,] <global2> [, <key2>], callback(<error>, <result>));
Example 1 (merge ^MyGlobal2 to ^MyGlobal1):
global1 = new mglobal(db, 'MyGlobal1');
global2 = new mglobal(db, 'MyGlobal2');
global1.merge(global2);
Example 2 (merge ^MyGlobal2(0) to ^MyGlobal1(1)):
global1 = new mglobal(db, 'MyGlobal1', 1);
global2 = new mglobal(db, 'MyGlobal2', 0);
global1.merge(global2);
Alternatively:
global1 = new mglobal(db, 'MyGlobal1');
global2 = new mglobal(db, 'MyGlobal2');
global1.merge(1, global2, 0);
<global>.reset(<global_name>[, <fixed_key>]);
Example:
// Process orders for customer #1
customer_orders = db.mglobal("Customer", 1, "orders")
do_work ...
// Process orders for customer #2
customer_orders.reset("Customer", 2, "orders");
do_work ...
This facility provides high-performance techniques for traversing records held in database globals.
The first task is to specify the 'query' for the global traverse.
query = new mcursor(db, {global: <global_name>, key: [<seed_key>]}[, <options>]);
Or:
query = db.mglobalquery({global: <global_name>, key: [<seed_key>]}[, <options>]);
The 'options' object can contain the following properties:
multilevel: A boolean value (default: multilevel: false). Set to 'true' to return all descendant nodes from the specified 'seed_key'.
getdata: A boolean value (default: getdata: false). Set to 'true' to return any data values associated with each global node returned.
format: Format for output (default: not specified). If the output consists of multiple data elements, the return value (by default) is a JavaScript object made up of a 'key' array and an associated 'data' value. Set to "url" to return such data as a single URL escaped string including all key values ('key[1->n]') and any associated 'data' value.
Example (return all keys and names from the 'Person' global):
query = db.mglobalquery({global: "Person", key: [""]}, {multilevel: false, getdata: true});
In key order:
result = query.next();
In reverse key order:
result = query.previous();
In all cases these methods will return 'null' when the end of the dataset is reached.
Example 1 (return all key values from the 'Person' global - returns a simple variable):
query = db.mglobalquery({global: "Person", key: [""]});
while ((result = query.next()) !== null) {
console.log("result: " + result);
}
Example 2 (return all key values and names from the 'Person' global - returns an object):
query = db.mglobalquery({global: "Person", key: [""]}, multilevel: false, getdata: true);
while ((result = query.next()) !== null) {
console.log("result: " + JSON.stringify(result, null, '\t'));
}
Example 3 (return all key values and names from the 'Person' global - returns a string):
query = db.mglobalquery({global: "Person", key: [""]}, multilevel: false, getdata: true, format: "url"});
while ((result = query.next()) !== null) {
console.log("result: " + result);
}
Example 4 (return all key values and names from the 'Person' global, including any descendant nodes):
query = db.mglobalquery({global: "Person", key: [""]}, {{multilevel: true, getdata: true});
while ((result = query.next()) !== null) {
console.log("result: " + JSON.stringify(result, null, '\t'));
}
query = db.mglobalquery({global: <seed_global_name>}, {globaldirectory: true});
Example (return all global names held in the current directory)
query = db.mglobalquery({global: ""}, {globaldirectory: true});
while ((result = query.next()) !== null) {
console.log("result: " + result);
}
In mg-dbx-bdb the default character encoding scheme is UTF-8. When transmitting binary data between the database and Node.js there are two options.
On the input (to the database) side all mg-dbx-bdb function arguments can be presented as Node.js Buffers and mg-dbx-bdb will automatically detect that an argument is a Buffer and process it accordingly.
On the retrieval side, the following functions can be used to return the output as a Node.js Buffer.
These functions work the same way as their non '_bx' suffixed counterparts. The only difference is that they will return data as a Node.js Buffer as opposed to a type of String.
The following two examples illustrate the two schemes for receiving binary data from the database.
Example 1: Receive binary data from a DB function as a Node.js 8-bit character stream
<db>.charset('ascii');
var stream_str8 = <db>.get(<key>);
<db>.charset('utf-8'); // reset character encoding
Example 2: Receive binary data from a DB record as a Node.js Buffer
var stream_buffer = <db>.get_bx(<key>);
mg-dbx-bdb functionality can be used with Node.js/V8 worker threads. This enhancement is available with Node.js v12 (and later).
Use the following constructs for instantiating mg-dbx-bdb objects in multi-threaded applications:
// Use:
var <global> = new mglobal(<db>, <global>);
// Instead of:
var <global> = <db>.mglobal(<global>);
// Use:
var <cursor> = new mcursor(<db>, <global_query>);
// Instead of:
var <cursor> = <db>.mglobalquery(<global_query>)
The following scheme illustrates how mg-dbx-bdb should be used in threaded Node.js applications.
const { Worker, isMainThread, parentPort, threadId } = require('worker_threads');
if (isMainThread) {
// start the threads
const worker1 = new Worker(__filename);
const worker2 = new Worker(__filename);
// process messages received from threads
worker1.on('message', (message) => {
console.log(message);
});
worker2.on('message', (message) => {
console.log(message);
});
} else {
var dbxbdb = require('mg-dbx-bdb').dbxbdb;
// And as required ...
var mglobal = require('mg-dbx-bdb').mglobal;
var mcursor = require('mg-dbx-bdb').mcursor;
var db = new dbxbdb();
db.open(<parameters>);
var global = new mglobal(db, <global>);
// do some work
var result = db.close();
// tell the parent that we're done
parentPort.postMessage("threadId=" + threadId + " Done");
}
mg-dbx-bdb provides an Event Log facility for recording errors in a physical file and, as an aid to debugging, recording the mg-dbx-bdb functions called by the application. This Log facility can also be used by Node.js applications.
To use this facility, the Event Log file must be specified using the following function:
db.setloglevel(<log_file>, <Log_level>, <log_filter>);
Where:
Examples:
db.setloglevel("c:/temp/mg-dbx-bdb.log", "e", "");
db.setloglevel("/tmp/mg-dbx-bdb.log", "ft", "dbx::set,mglobal::set,mcursor::execute");
Node.js applications can write their own messages to the Event Log using the following function:
db.logmessage(<message>, <title>);
Logging can be switched off by calling the setloglevel function without specifying a log level. For example:
db.setloglevel("c:/temp/mg-dbx-bdb.log");
Copyright (c) 2018-2024 MGateway Ltd,
Surrey UK.
All rights reserved.
http://www.mgateway.com
Email: cmunt@mgateway.com
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
FAQs
High speed Synchronous and Asynchronous access to Berkeley DB and LMDB from Node.js.
We found that mg-dbx-bdb demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket’s threat research team has detected six malicious npm packages typosquatting popular libraries to insert SSH backdoors.
Security News
MITRE's 2024 CWE Top 25 highlights critical software vulnerabilities like XSS, SQL Injection, and CSRF, reflecting shifts due to a refined ranking methodology.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.