jsonblite

JSONBLite: Single-file binary JSON database

JSONBLite is a single file, key-value binary JSON database, implemented as a TypeScript class for Node.js. A naive solution for persistent JSON storage, embeddable in Node.js applications. Operations are synchronous, and writes are serialized with file locks plus a write-ahead journal.

JSONBLite uses CBOR (Concise Binary Object Representation) standard to store binary JSON data. It's more compact and faster to parse than JSON. Any JSON data can be encoded/decoded in the database.

The index is a serialized JavaScript Map, not a tree on disk. The index is initialized in memory in its entirety as a Map to allow for fast lookups in memory.

See jsonblite-example for LIVE DEMO of a simple server application running JSONBLite.

Warning: Not recommended for any use in its current state. Expect data loss and corruption.

Features

• Single file database
• Single TypeScript class in less than 500 lines of code
• CBOR for binary JSON storage
• File locks and journaling for crash recovery
• In-memory Map index
• Append-only log, with a manual vacuum() garbage collection

Dependencies

• cbor-x for CBOR encoding/decoding
• fs-ext for flock implementation

Usage

Install the package from npm

npm i jsonblite

Use the class in your Node.js application

import JSONBLite from 'jsonblite';

// initialize JSONBlite instance by reading or creating a database file
const db = new JSONBLite('./data/db.jsonblite');

db.write('k', { value: 'Hello, world!', number: 1 });
db.write('k2', 123);
db.read('k');
// -> { value: 'Hello, world!', number: 1 }
const dumpStream = db.dump();
dumpStream.pipe(process.stdout);
db.dump('./data/db.json');
db.delete('k2');
db.read('k');
// -> null
db.keys();
// -> [ 'k' ]

API

Constructor

• new JSONBLite(filename: string, options?: { verbose: boolean }): Create a new database instance and file.

Methods

• read(key: string): Read a value from the database.
• write(key: string, value: any): Write a value to the database.
• delete(key: string): Delete a value from the database index.
• keys(): Read all keys from the database.

Maintenance

• vacuum(): Run to permanently remove deleted data from the database file, and compact the file.
• dump(): Return a readable stream with a consistent JSON snapshot.
• dump(filename: string): Write a consistent JSON snapshot to a file.

File Format

+-----------------+
| Header (fixed)  | 6A73 6F6E 626C 6974 6501 1C00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 00a3
+-----------------+
| Data   (varlen) | 8261 6B76 616C 7565 6A48 656C 6C6F 2C20 776F 726C 6421 6963 6E75 6D31 6A62 6F6F 6C74 ...
+-----------------+
| Index  (varlen) | 646b 6579 3182 185c 0d64 6b65 7932 8218 5c0d 646b 6579 3382 185c 0d64 6b65 7934 8218 ...
+-----------------+

Header

Header is a fixed 36-byte structure at the beginning of the file.

Field	Size	Description
magic	9 bytes	Magic number (0x6A736F6E626C6974, "jsonblite")
version	1 byte	Version number (0x01)
index_size	4 bytes	Index size (uint)
data_size	6 bytes	Data size
last_modified	8 bytes	Unix timestamp of last modification
last_vacuum	8 bytes	Unix timestamp of last vacuum

Index

In-memory, index is a variable length JavaScript Map of keys to record data [offset, size]. On disk, it's a CBOR-encoded Map.

Field	Type	Description
key	string	any string
offset	uint	Location of the data record offset
size	uint	Size of the data record in bytes

Data

Data is saved as a log of CBOR-encoded JSON records. Data is accessed by the offsets in the index.

Dump

dump() produces JSON with meta and data fields. The same payload is streamed to the caller or written to a file when a filename is provided.

{
    "meta": {
        "version": 1,
        "data_size": 48,
        "index_size": 28,
        "last_vacuum": 0
    },
    "data": { 
        "key": { "value": "Hello, world!", "num": 1, "bool": true },
        "key2": { "value": "Example", "bool": false }
     },
}