@harperfast/rocksdb-js

rocksdb-js

A Node.js binding for the RocksDB library.

Features

• Supports optimistic and pessimistic transactions
• Hybrid sync/async data retrieval
• Range queries return an iterable with array-like methods and lazy evaluation
• Transaction log system for recording transaction related data
• Custom stores provide ability to override default database interactions
• Efficient binary key and value encoding
• Access to internal RocksDB statistics
• Designed for Node.js and Bun on Linux, macOS, and Windows

Example

const db = RocksDatabase.open('/path/to/db');

for (const key of ['a', 'b', 'c', 'd', 'e']) {
	await db.put(key, `value ${key}`);
}

console.log(await db.get('b')); // `value b`

for (const { key, value } of db.getRange({ start: 'b', end: 'd' })) {
	console.log(`${key} = ${value}`);
}

await db.transaction(async (txn: Transaction) => {
	await txn.put('f', 'value f');
	await txn.remove('c');
});

Usage

new RocksDatabase(path, options?)

Creates a new database instance.

• path: string The path to write the database files to. This path does not need to exist, but the parent directories do.
• options: object [optional]
  • disableWAL: boolean Whether to disable the RocksDB write ahead log. Defaults to false.
  • enableStats: boolean When true and the database is open, RocksDB will captures stats that are retrieved by calling db.getStats(). Enabling statistics imposes 5-10% in overhead. Defaults to false.
  • name: string The column family name. Defaults to "default".
  • noBlockCache: boolean When true, disables the block cache. Block caching is enabled by default and the cache is shared across all database instances.
  • parallelismThreads: number The number of background threads to use for flush and compaction. Defaults to 1.
  • pessimistic: boolean When true, throws conflict errors when they occur instead of waiting until commit. Defaults to false.
  • readOnly: boolean When true, the database is opened in read-only mode. Read operations are permitted. Write operations will throw an error with code ERR_DATABASE_READONLY. Transactions are a no-op in read-only mode.
  • statsLevel: StatsLevel Controls which type of statistics to skip and reduce statistic overhead. Defaults to StatsLevel.ExceptDetailedTimers.
  • store: Store A custom store that handles all interaction between the RocksDatabase or Transaction instances and the native database interface. See Custom Store for more information.
  • transactionLogMaxAgeThreshold: number The threshold for the transaction log file's last modified time to be older than the retention period before it is rotated to the next sequence number. Value must be between 0.0 and 1.0. A threshold of 0.0 means ignore age check. Defaults to 0.75.
  • transactionLogMaxSize: number The maximum size of a transaction log file. If a log file is empty, the first log entry will always be added regardless if it's larger than the max size. If a log file is not empty and the entry is larger than the space available, the log file is rotated to the next sequence number. Defaults to 16 MB.
  • transactionLogRetention: string | number The number of minutes to retain transaction logs before purging. Defaults to '3d' (3 days).
  • transactionLogsPath: string The path to store transaction logs. Defaults to "${db.path}/transaction_logs".

db.close()

Closes a database. This function can be called multiple times and will only close an opened database. A database instance can be reopened once its closed.

const db = RocksDatabase.open('foo');
db.close();

db.columns: string[]

Returns the list of column families in the RocksDB database.

const db = RocksDatabase.open('path/to/db');
console.log(db.columns); // ['default']

const db2 = new RocksDatabase('path/to/db', { name: 'users' });
console.log(db.columns); // ['default', 'users']

db.config(options)

Sets global database settings.

• options: object
  • blockCacheSize: number The amount of memory in bytes to use to cache uncompressed blocks. Defaults to 32MB. Set to 0 (zero) disables block cache for future opened databases. Existing block cache for any opened databases is resized immediately. Negative values throw an error.

RocksDatabase.config({
	blockCacheSize: 100 * 1024 * 1024, // 100MB
});

db.isOpen(): boolean

Returns true if the database is open, otherwise false.

console.log(db.isOpen()); // true or false

db.name: string

Returns the database column family's name.

const db = new RocksDatabase('path/to/db');
console.log(db.name); // 'default'

const db2 = new RocksDatabase('path/to/db', { name: 'users' });
console.log(db.name); // 'users'

db.open(): RocksDatabase

Opens the database at the given path. This must be called before performing any data operations.

import { RocksDatabase } from '@harperfast/rocksdb-js';

const db = new RocksDatabase('path/to/db');
db.open();

There's also a static open() method for convenience that performs the same thing:

const db = RocksDatabase.open('path/to/db');

db.status: 'opened' | 'closed'

Returns a string 'opened' or 'closed' indicating if the database is opened or closed.

console.log(db.status);

Data Operations

db.clear(options?): Promise<number>

Asychronously removes all data in the current database.

• options: object
  • batchSize?: number The number of records to remove at once. Defaults to 10000.

Returns the number of entries that were removed.

Note: This does not remove data from other column families within the same database path.

for (let i = 0; i < 10; i++) {
	db.putSync(`key${i}`, `value${i}`);
}
const entriesRemoved = await db.clear();
console.log(entriesRemoved); // 10

db.clearSync(options?): number

Synchronous version of db.clear().

• options: object
  • batchSize?: number The number of records to remove at once. Defaults to 10000.

for (let i = 0; i < 10; i++) {
	db.putSync(`key${i}`, `value${i}`);
}
const entriesRemoved = db.clearSync();
console.log(entriesRemoved); // 10

db.destroy(): void

Completely removes a database based on the db instance's path including all data, column families, and files on disk.

db.destroy();
console.log(fs.existsSync(db.path)); // false

db.drop(): Promise<void>

Removes all entries in the database. If the database was opened with a name, the database will be deleted on close.

const db = RocksDatabase.open('path/to/db', { name: 'users' });
await db.drop();
db.close();

db.dropSync(): void

Synchronous version of db.drop().

const db = RocksDatabase.open('path/to/db');
db.dropSync();
db.close();

db.get(key: Key, options?: GetOptions): MaybePromise<any>

Retreives the value for a given key. If the key does not exist, it will resolve undefined.

const result = await db.get('foo');
assert.equal(result, 'foo');

If the value is in the memtable or block cache, get() will immediately return the value synchronously instead of returning a promise.

const result = db.get('foo');
const value = result instanceof Promise ? await result : result;
assert.equal(result, 'foo');

Note that all errors are returned as rejected promises.

db.getSync(key: Key, options?: GetOptions): any

Synchronous version of get().

db.getKeys(options?: IteratorOptions): ExtendedIterable

Retrieves all keys within a range.

for (const key of db.getKeys()) {
	console.log(key);
}

db.getKeysCount(options?: RangeOptions): number

Retrieves the number of keys within a range.

const total = db.getKeysCount();
const range = db.getKeysCount({ start: 'a', end: 'z' });

db.getMonotonicTimestamp(): number

Returns the current timestamp as a monotonically increasing timestamp in milliseconds represented as a decimal number.

const ts = db.getMonotonicTimestamp();
console.log(ts); // 1764307857213.739

db.getOldestSnapshotTimestamp(): number

Returns a number representing a unix timestamp of the oldest unreleased snapshot.

Snapshots are only created during transactions. When the database is opened in optimistic mode (the default), the snapshot will be created on the first read. When the database is opened in pessimistic mode, the snapshot will be created on the first read or write.

console.log(db.getOldestSnapshotTimestamp()); // returns `0`, no snapshots

const promise = db.transaction(async (txn) => {
	// perform a write to create a snapshot
	await txn.get('foo');
	await setTimeout(100);
});

console.log(db.getOldestSnapshotTimestamp()); // returns `1752102248558`

await promise;
// transaction completes, snapshot released

console.log(db.getOldestSnapshotTimestamp()); // returns `0`, no snapshots

db.getDBProperty(propertyName: string): string | undefined

Gets a RocksDB database property as a string.

• propertyName: string The name of the property to retrieve (e.g., ) 'rocksdb.levelstats'.

Returns undefined if the property is not found.

const db = RocksDatabase.open('/path/to/database');
const levelStats = db.getDBProperty('rocksdb.levelstats');
const stats = db.getDBProperty('rocksdb.stats');

db.getDBIntProperty(propertyName: string): number | undefined

Gets a RocksDB database property as an integer.

• propertyName: string The name of the property to retrieve (e.g., ) 'rocksdb.num-blob-files'.

Returns undefined if the property is not found.

const db = RocksDatabase.open('/path/to/database');
const blobFiles = db.getDBIntProperty('rocksdb.num-blob-files');
const numKeys = db.getDBIntProperty('rocksdb.estimate-num-keys');

db.getRange(options?: IteratorOptions): ExtendedIterable

Retrieves a range of keys and their values. Supports both synchronous and asynchronous iteration.

// sync
for (const { key, value } of db.getRange()) {
	console.log({ key, value });
}

// async
for await (const { key, value } of db.getRange()) {
	console.log({ key, value });
}

// key range
for (const { key, value } of db.getRange({ start: 'a', end: 'z' })) {
	console.log({ key, value });
}

db.getUserSharedBuffer(key: Key, defaultBuffer: ArrayBuffer, options?)

Creates a new buffer with the contents of defaultBuffer that can be accessed across threads. This is useful for storing data such as flags, counters, or any ArrayBuffer-based data.

• options?: object
  • callback?: () => void A optional callback is called when notify() on the returned buffer is called.

Returns a new ArrayBuffer with two additional methods:

• notify() - Invokes the options.callback, if specified.
• cancel() - Removes the callback; future notify() calls do nothing

Note: If a shared buffer already exists for the given key, the returned ArrayBuffer will reference this existing shared buffer. Once all ArrayBuffer instances have gone out of scope and garbage collected, the underlying memory and notify callback will be freed.

const buffer = new Uint8Array(db.getUserSharedBuffer('isDone', new ArrayBuffer(1)));
done[0] = 0;

if (done[0] !== 1) {
	done[1] = 1;
}

const incrementer = new BigInt64Array(
	db.getUserSharedBuffer('next-id', new BigInt64Array(1).buffer)
);
incrementer[0] = 1n;

function getNextId() {
	return Atomics.add(incrementer, 0, 1n);
}

db.put(key: Key, value: any, options?: PutOptions): Promise

Stores a value for a given key.

await db.put('foo', 'bar');

db.putSync(key: Key, value: any, options?: PutOptions): void

Synchronous version of put().

db.remove(key: Key): Promise

Removes the value for a given key.

await db.remove('foo');

db.removeSync(key: Key): void

Synchronous version of remove().

Transactions

db.transaction<T>(callback: TransactionCallback<T>, options?: TransactionOptions): Promise<T>

Executes all database operations within the specified callback within a single transaction. If the callback completes without error, the database operations are automatically committed. However, if an error is thrown during the callback, all database operations will be rolled back.

import type { Transaction } from '@harperfast/rocksdb-js';
await db.transaction(async (txn: Transaction) => {
	await txn.put('foo', 'baz');
});

Additionally, you may pass the transaction into any database data method:

await db.transaction(async (transaction: Transaction) => {
	await db.put('foo', 'baz', { transaction });
});

Note that db.transaction() returns whatever value the transaction callback returns:

const isBar = await db.transaction(async (txn: Transaction) => {
	const foo = await txn.get('foo');
	return foo === 'bar';
});
console.log(isBar ? 'Foo is bar' : 'Foo is not bar');

db.transactionSync<T>(callback: TransactionCallback<T>, options?: TransactionOptions): T

Executes a transaction callback and commits synchronously. Once the transaction callback returns, the commit is executed synchronously and blocks the current thread until finished.

Inside a synchronous transaction, use getSync(), putSync(), and removeSync().

import type { Transaction } from '@harperfast/rocksdb-js';
db.transactionSync((txn: Transaction) => {
	txn.putSync('foo', 'baz');
});

Optimistic and Pessimistic Modes

rocksdb-js supports two different transaction modes: optimistic and pessimistic. The default mode is optimistic.

• Optimistic: Conflicts detected at commit time.
• Pessimistic: Conflicts throw immediately on detection.

When a database is opened in optimistic mode, transactions are not locked and can be retried if they fail with a conflict. When a database is opened in pessimistic mode, transactions are aborted and cannot be retried if they fail with a conflict.

Optimistic mode is the default mode and is recommended for most use cases. Pessimistic mode is recommended for use cases where you need to know immediately if a conflict occurs.

If a database is opened in one mode, it cannot be opened in a different mode. An error will be thrown when trying to open it in a different mode without closing the database first.

TransactionCallback<T>

(txn: Transaction, attempt: number) => T | PromiseLike<T>

A sync or async function to encapsulate all of the transaction operations. Once the function is executed, the transaction is automatically committed. If the function returns a value, it will be returned from the transaction call.

The txn parameter is a Transaction. See the Transaction section for more details.

The attempt parameter is the number of times the transaction has been retried.

TransactionOptions

• disableSnapshot?: boolean Whether to disable snapshots. Defaults to false.
• maxRetries?: number The maximum number of times to retry the transaction. Defaults to 3.
• retryOnBusy?: boolean Whether to retry the transaction if the commit fails with IsBusy. Defaults to true when the transaction is bound to a transaction log, otherwise false.

Transaction Retry Logic

The retry mechanism will only be active when the retryOnBusy option is true or when retryOnBusy is undefined and the transaction is bound to a transaction log. The attempts starts at 1 and ends at maxRetries.

When using a transaction log and the commit fails with ERR_BUSY, the transaction log will be in a bad state and the transaction will need to be retried. If the max retries is reached or the transaction is not retried, a ERR_TRANSACTION_ABANDONED error will be thrown.

Users should use the attempt transaction callback parameter to ensure duplicate transaction log entries are not added.

Class: Transaction

The transaction callback is passed in a Transaction instance which contains all of the same data operations methods as the RocksDatabase instance plus:

• txn.abort() Rolls back and closes the transaction. This method is automatically called after the transaction callback returns, so you shouldn't need to call it, but it's ok to do so. Once called, no further transaction operations are permitted. Calling this method multiple times has no effect.
• txn.commit(): Promise<void> Asynchronously commits the transaction and closes the transaction.
• txn.commitSync() Synchronously commits and closes the transaction.
• txn.getTimestamp(): number Retrieves the transaction start timestamp in seconds as a decimal. It defaults to the time at which the transaction was created.
• txn.id: number The read-only transaction ID. Transaction IDs are unique to the RocksDB database path, regardless the database name/column family.
• txn.setTimestamp(ts?: number): void Overrides the transaction start timestamp. If called without a timestamp, it will set the timestamp to the current time. The value must be in seconds with higher precision in the decimal.

txn.abort(): void

Rolls back and closes the transaction. This method is automatically called after the transaction callback returns, so you shouldn't need to call it, but it's ok to do so. Once called, no further transaction operations are permitted.

txn.commit(): Promise<void>

Commits and closes the transaction. This is a non-blocking operation and runs on a background thread. Once called, no further transaction operations are permitted.

txn.commitSync(): void

Synchronously commits and closes the transaction. This is a blocking operation on the main thread. Once called, no further transaction operations are permitted.

txn.getTimestamp(): number

Retrieves the transaction start timestamp in seconds as a decimal. It defaults to the time at which the transaction was created.

txn.id

Type: number

The transaction ID represented as a 32-bit unsigned integer. Transaction IDs are unique to the RocksDB database path, regardless the database name/column family.

txn.setTimestamp(ts: number?): void

Overrides the transaction start timestamp. If called without a timestamp, it will set the timestamp to the current time. The value must be in seconds with higher precision in the decimal.

await db.transaction(async (txn) => {
	txn.setTimestamp(Date.now() / 1000);
});

Events

Event: 'aftercommit'

The 'aftercommit' event is emitted after a transaction has been committed and the transaction has completed including waiting for the async worker thread to finish.

• result: object
  • next: null
  • last: null
  • txnId: number The id of the transaction that was just committed.

Event: 'beforecommit'

The 'beforecommit' event is emitted before a transaction is about to be committed.

Event: 'begin-transaction'

The 'begin-transaction' event is emitted right before the transaction function is executed.

Event: 'committed'

The 'committed' event is emitted after the transaction has been written. When this event is emitted, the transaction is still cleaning up. If you need to know when the transaction is fully complete, use the 'aftercommit' event.

Event API

rocksdb-js provides a EventEmitter-like API that lets you asynchronously notify events to one or more synchronous listener callbacks. Events are scoped by database path.

Unlike EventEmitter, events are emitted asynchronously, but in the same order that the listeners were added.

const callback = (name) => console.log(`Hi from ${name}`);
db.addListener('foo', callback);
db.notify('foo');
db.notify('foo', 'bar');
db.removeListener('foo', callback);

addListener(event: string, callback: () => void): void

Adds a listener callback for the specific key.

db.addListener('foo', () => {
	// this callback will be executed asynchronously
});

db.addListener(1234, (...args) => {
	console.log(args);
});

listeners(event: string): number

Gets the number of listeners for the given key.

db.listeners('foo'); // 0
db.addListener('foo', () => {});
db.listeners('foo'); // 1

on(event: string, callback: () => void): void

Alias for addListener().

once(event: string, callback: () => void): void

Adds a one-time listener, then automatically removes it.

db.once('foo', () => {
	console.log('This will only ever be called once');
});

removeListener(event: string, callback: () => void): boolean

Removes an event listener. You must specify the exact same callback that was used in addListener().

const callback = () => {};
db.addListener('foo', callback);

db.removeListener('foo', callback); // return `true`
db.removeListener('foo', callback); // return `false`, callback not found

off(event: string, callback: () => void): boolean

Alias for removeListener().

notify(event: string, ...args?): boolean

Call all listeners for the given key. Returns true if any callbacks were found, otherwise false.

Unlike EventEmitter, events are emitted asynchronously, but in the same order that the listeners were added.

You can optionally emit one or more arguments. Note that the arguments must be serializable. In other words, undefined, null, strings, booleans, numbers, arrays, and objects are supported.

db.notify('foo');
db.notify(1234);
db.notify({ key: 'bar' }, { value: 'baz' });

Statistics

Retrieve RocksDB statistics at runtime. You must set enableStats: true when calling db.open(). Statistics are captured at the database level and include all column families.

RocksDB has two types of statistics: tickers and histograms. Tickers are 64-bit unsigned integers that measure counters. Histograms are objects containing various measurements of statistic distribution across all operations.

import { RocksDatabase, stats } from '@harperfast/rocksdb-js';
const db = RocksDatabase.open('/path/to/db', {
	enableStats: true,
	statsLevel: stats.StatsLevel.ExceptDetailedTimers, // default
});
console.log(db.getStats());

db.getStat(statName: string): RocksDBStat

Retrieves a single statistic value. Return value is either a number or StatsHistogramData object.

console.log(db.getStat('rocksdb.block.cache.miss'));

db.getStats(all?: boolean): RocksDBStats

Returns an object containing a curated list of column family-level properties, internal tickers stats, and internal histogram stats. Return value is an object with the stat name as the key and a RocksDBStat as the value.

By default, it only returns the most meaningful internal stats. When all = true, it returns the same column family-level properties, but includes all internal tickers and histogram stats.

Column family and ticker stat values are 64-bit unsigned integers and histogram values are StatsHistogramData objects.

// get essential stats
console.log(db.getStats());

// get all stats
console.log(db.getStats(true));

stats

An object containing stat specific constants including ticker and histogram names.

stats.histograms

An array of internal histogram stat names.

import { stats } from '@harperfast/rocksdb-js';
console.log(stats.histograms);

stats.tickers

An array of internal ticker stat names.

import { stats } from '@harperfast/rocksdb-js';
console.log(stats.tickers);

stats.StatsLevel

The stats.StatsLevel contains constants used to set which types of skip and reduce statistic overhead.

• StatsLevel.DisableAll Disable all metrics.
• StatsLevel.ExceptTickers Disable all tickers.
• StatsLevel.ExceptHistogramOrTimers Disable timer stats and skip histogram stats.
• StatsLevel.ExceptTimers Skip timer stats
• StatsLevel.ExceptDetailedTimers Skip time waiting for mutex locks and compression.
• StatsLevel.ExceptTimeForMutex Skip time waiting for mutex locks.
• StatsLevel.All Collects all stats.

type RocksDBStat = number | StatsHistogramData

A RocksDBStat is either a number or StatsHistogramData object.

type RocksDBStats = Record<string, RocksDBStat>

A RocksDBStats is an object with the stat name as the key and a RocksDBStat as the value.

type StatsHistogramData

An object is a record with the following properties:

• average: number A double containing the average value.
• count: number An unsigned 64-bit integer containing the number of values.
• max: number A double containing the maximum value.
• median: number A double containing the median value.
• min: number A double containing the minimum value.
• percentile95: number A double containing the 95th percentile value.
• percentile99: number A double containing the 99th percentile value.
• standardDeviation: number A double containing the standard deviation.
• sum: number An unsigned 64-bit integer containing the sum of all values.

Exclusive Locking

rocksdb-js includes a handful of functions for executing thread-safe mutually exclusive functions.

db.hasLock(key: Key): boolean

Returns true if the database has a lock for the given key, otherwise false.

db.hasLock('foo'); // false
db.tryLock('foo'); // true
db.hasLock('foo'); // true

db.tryLock(key: Key, onUnlocked?: () => void): boolean

Attempts to acquire a lock for a given key. If the lock is available, the function returns true and the optional onUnlocked callback is never called. If the lock is not available, the function returns false and the onUnlocked callback is queued until the lock is released.

When a database is closed, all locks associated to it will be unlocked.

db.tryLock('foo', () => {
	console.log('never fired');
}); // true, callback ignored

db.tryLock('foo', () => {
	console.log('hello world');
}); // false, already locked, callback queued

db.unlock('foo'); // fires second lock callback

The onUnlocked callback function can be used to signal to retry acquiring the lock:

function doSomethingExclusively() {
	// if lock is unavailable, queue up callback to recursively retry
	if (db.tryLock('foo', () => doSomethingExclusively())) {
		// lock acquired, do something exclusive

		db.unlock('foo');
	}
}

db.unlock(key): boolean

Releases the lock on the given key and calls any queued onUnlocked callback handlers. Returns true if the lock was released or false if the lock did not exist.

db.tryLock('foo');
db.unlock('foo'); // true
db.unlock('foo'); // false, already unlocked

db.withLock(key: Key, callback: () => void | Promise<void>): Promise<void>

Runs a function with guaranteed exclusive access across all threads.

await db.withLock('key', async () => {
	// do something exclusive
	console.log(db.hasLock('key')); // true
});

If there are more than one simultaneous lock requests, it will block them until the lock is available.

await Promise.all([
	db.withLock('key', () => {
		console.log('first lock blocking for 100ms');
		return new Promise((resolve) => setTimeout(resolve, 100));
	}),
	db.withLock('key', () => {
		console.log('second lock blocking for 100ms');
		return new Promise((resolve) => setTimeout(resolve, 100));
	}),
	db.withLock('key', () => {
		console.log('third lock acquired');
	}),
]);

Note: If the callback throws an error, Node.js suppress the error. Node.js 18.3.0 introduced a --force-node-api-uncaught-exceptions-policy flag which will cause errors to emit the 'uncaughtException' event. Future Node.js releases will enable this flag by default.

db.flush(): Promise<void>

Flushes all in-memory data to disk asynchronously.

await db.flush();

db.flushSync(): void

Flushes all in-memory data to disk synchronously. Note that this can be an expensive operation, so it is recommended to use flush() if you want to keep the event loop free.

db.flushSync();

Transaction Log

A user controlled API for logging transactions. This API is designed to be generic so that you can log gets, puts, and deletes, but also arbitrary entries.

Transaction logs are isolated by the database path allowing different column families in the same database to share the transaction log store, but not other databases.

db.listLogs(): string[]

Returns an array of log store names.

const names = db.listLogs();

db.purgeLogs(options?): string[]

Deletes transaction log files older than the transactionLogRetention (defaults to 3 days).

• options: object
  • before?: number Remove all transaction log files older than the specified timestamp.
  • destroy?: boolean When true, deletes transaction log stores including all log sequence files on disk.
  • name?: string The name of a store to limit the purging to.

Returns an array with the full path of each log file deleted.

const removed = db.purgeLogs();
console.log(`Removed ${removed.length} log files`);

db.useLog(name): TransactionLog

Gets or creates a TransactionLog instance. Internally, the TransactionLog interfaces with a shared transaction log store that is used by all threads. Multiple worker threads can use the same log at the same time.

• name: string | number The name of the log. Numeric log names are converted to a string.

const log1 = db.useLog('foo');
const log2 = db.useLog('foo'); // gets existing instance (e.g. log1 === log2)
const log3 = db.useLog(123);

Transaction instances also provide a useLog() method that binds the returned transaction log to the transaction so you don't need to pass in the transaction id every time you add an entry.

await db.transaction(async (txn) => {
	const log = txn.useLog('foo');
	log.addEntry(Buffer.from('hello'));
});

Class: TransactionLog

A TransactionLog lets you add arbitrary data bound to a transaction that is automatically written to disk right before the transaction is committed. You may add multiple enties per transaction. The underlying architecture is thread safe.

• log.addEntry()
• log.path
• log.query()

log.addEntry(data, transactionId): void

Adds an entry to the transaction log.

• data: Buffer | UInt8Array The entry data to store. There is no inherent limit beyond what Node.js can handle.
• transactionId: Number A related transaction used to batch entries on commit.

const log = db.useLog('foo');
await db.transaction(async (txn) => {
	log.addEntry(Buffer.from('hello'), txn.id);
});

If using txn.useLog() (instead of db.useLog()), you can omit the transaction id from addEntry() calls.

await db.transaction(async (txn) => {
	const log = txn.useLog('foo');
	log.addEntry(Buffer.from('hello'));
});

Note that the TransactionLog class also has internal methods _getMemoryMapOfFile, _findPosition, and _getLastCommittedPosition that should not be used directly and may change in any version.

log.path: string

Returns the path to the transaction log store files.

const log = db.useLog('foo');
console.log(log.path);

log.query(options?): IterableIterator<TransactionLogEntry>

Returns an iterable/iterator that streams all log entries for the given filter.

• options: object
  • start?: number The transaction start timestamp.
  • end?: string The transction end timestamp.
  • exclusiveStart?: boolean When true, this will only match transactions with timestamps after the start timestamp.
  • exactStart?: boolean When true, this will only match and iterate starting from a transaction with the given start timestamp. Once the specified transaction is found, all subsequent transactions will be returned (regardless of whether their timestamp comes before the start time). This can be combined with exactStart, finding the specified transaction, and returning all transactions that follow. By default, all transactions equal to or greater than the start timestamp will be included.
  • readUncommitted?: boolean When true, this will include uncommitted transaction entries. Normally transaction entries that haven't finished committed are not included. This is particularly useful for replaying transaction logs on startup where many entries may have been written to the log but are no longer considered committed if they were not flushed to disk.
  • startFromLastFlushed?: boolean When true, this will only match transactions that have been flushed from RocksDB's memtables to disk (and are within any provided start and end filters, if included). This is useful for replaying transaction logs on startup where many entries may have been written to the log but are no longer considered committed if they were not flushed to disk.

The iterator produces an object with the log entry timestamp and data.

• object
  • data: Buffer The entry data.
  • timestamp: number The entry timestamp used to collate entries by transaction.
  • endTxn: boolean This is true when the entry is the last entry in a transaction.

const log = db.useLog('foo');
const iter = log.query({});
for (const entry of iter) {
	console.log(entry);
}

const lastHour = Date.now() - 60 * 60 * 1000;
const rangeIter = log.query({ start: lastHour, end: Date.now() });
for (const entry of rangeIter) {
	console.log(entry.timestamp, entry.data);
}

log.getLogFileSize(sequenceNumber?: number): number

Returns the size of the given transaction log sequence file in bytes. Omit the sequence number to get the total size of all the transaction log sequence files for this log.

Transaction Log Parser

parseTransactionLog(file)

In general, you should use log.query() to query the transaction log, however, if you need to load an entire transaction log into memory and want detailed information about entries, you can use the parseTransactionLog() utility function.

const everything = parseTransactionLog('/path/to/file.txnlog');
console.log(everything);

Returns an object containing all of the information in the log file.

• size: number The size of the file.
• version: number The log file format version.
• entries: LogEntry[] An array of transaction log entries.
  • data: Buffer The entry data.
  • flags: number Transaction related flags.
  • length: number The size of the entry data.
  • timestamp: number The entry timestamp.

currentThreadId(): number

Returns the current thread ID.

import { currentThreadId } from '@harperfast/rocksdb-js';
console.log(currentThreadId());

registryStatus(): RegistryStatus

Returns an array containing that status of all active RocksDB instances.

• path: string The database path.
• refCount: number The number of JavaScript database instances plus the registry's reference.
• columnFamiles: object A map of column family names and their their info.
  • userSharedBuffers: number The count of active user shared buffers.
• transactions: number The count of active transactions.
• closables: number The count of active database, transactions, and iterators.
• locks: number The count of active locks.
• listenerCallbacks: number The count of in-flight callbacks.

import { registryStatus } from '@harperfast/rocksdb-js';
console.log(registryStatus());

shutdown(): void

The shutdown() will flush all in-memory data to disk and wait for any outstanding compactions to finish, for all open databases. It is highly recommended to call this in a process exit event listener (on the main thread), to ensure that all data is flushed to disk before the process exits:

import { shutdown } from '@harperfast/rocksdb-js';
process.on('exit', shutdown);

versions: { 'rocksdb': string; 'rocksdb-js': string }

Returns the rocksdb-js and RocksDB version.

import { versions } from '@harperfast/rocksdb-js';
console.log(versions); // { "rocksdb": "10.10.1", "rocksdb-js": "0.1.2" }

Custom Store

The store is a class that sits between the RocksDatabase or Transaction instance and the native RocksDB interface. It owns the native RocksDB instance along with various settings including encoding and the db name. It handles all interactions with the native RocksDB instance.

The default Store contains the following methods which can be overridden:

• constructor(path, options?)
• close()
• decodeKey(key)
• decodeValue(value)
• encodeKey(key)
• encodeValue(value)
• get(context, key, resolve, reject, txnId?)
• getCount(context, options?, txnId?)
• getKeys(context, options?)
• getKeysCount(context, options?)
• getRange(context, options?)
• getSync(context, key, options?)
• getUserSharedBuffer(key, defaultBuffer?)
• hasLock(key)
• isOpen()
• listLogs()
• open()
• putSync(context, key, value, options?)
• removeSync(context, key, options?)
• tryLock(key, onUnlocked?)
• unlock(key)
• useLog(context, name)
• withLock(key, callback?)

To use it, extend the default Store and pass in an instance of your store into the RocksDatabase constructor.

import { RocksDatabase, Store } from '@harperfast/rocksdb-js';

class MyStore extends Store {
  get(context, key, resolve, reject, txnId) {
    console.log('Getting:' key);
    return super.get(context, key, resolve, reject, txnId);
  }

  putSync(context, key, value, options) {
    console.log('Putting:', key);
    return super.putSync(context, key, value, options);
  }
}

const myStore = new MyStore('path/to/db');
const db = RocksDatabase.open(myStore);
await db.put('foo', 'bar');
console.log(await db.get('foo'));

[!IMPORTANT] If your custom store overrides putSync() without calling super.putSync() and it performs its own this.encodeKey(key), then you MUST encode the VALUE before you encode the KEY.

Keys are encoded into a shared buffer. If the database is opened with the sharedStructuresKey option, encoding the value will load and save the structures which encodes the sharedStructuresKey overwriting the encoded key in the shared key buffer, so it's ultra important that you encode the value first!

Interfaces

RocksDBOptions

• options: object
  • adaptiveReadahead: boolean When true, RocksDB will do some enhancements for prefetching the data. Defaults to true. Note that RocksDB defaults this to false.
  • asyncIO: boolean When true, RocksDB will prefetch some data async and apply it if reads are sequential and its internal automatic prefetching. Defaults to true. Note that RocksDB defaults this to false.
  • autoReadaheadSize: boolean When true, RocksDB will auto-tune the readahead size during scans internally based on the block cache data when block caching is enabled, an end key (e.g. upper bound) is set, and prefix is the same as the start key. Defaults to true.
  • backgroundPurgeOnIteratorCleanup: boolean When true, after the iterator is closed, a background job is scheduled to flush the job queue and delete obsolete files. Defaults to true. Note that RocksDB defaults this to false.
  • fillCache: boolean When true, the iterator will fill the block cache. Filling the block cache is not desirable for bulk scans and could impact eviction order. Defaults to false. Note that RocksDB defaults this to true.
  • readaheadSize: number The RocksDB readahead size. RocksDB does auto-readahead for iterators when there is more than two reads for a table file. The readahead starts at 8KB and doubles on every additional read up to 256KB. This option can help if most of the range scans are large and if a larger readahead than that enabled by auto-readahead is needed. Using a large readahead size (> 2MB) can typically improve the performance of forward iteration on spinning disks. Defaults to 0.
  • tailing: boolean When true, creates a "tailing iterator" which is a special iterator that has a view of the complete database including newly added data and is optimized for sequential reads. This will return records that were inserted into the database after the creation of the iterator. Defaults to false.

RangeOptions

Extends RocksDBOptions.

• options: object
  • end: Key | Uint8Array The range end key, otherwise known as the "upper bound". Defaults to the last key in the database.
  • exclusiveStart: boolean When true, the iterator will exclude the first key if it matches the start key. Defaults to false.
  • inclusiveEnd: boolean When true, the iterator will include the last key if it matches the end key. Defaults to false.
  • start: Key | Uint8Array The range start key, otherwise known as the "lower bound". Defaults to the first key in the database.

IteratorOptions

Extends RangeOptions.

• options: object
  • reverse: boolean When true, the iterator will iterate in reverse order. Defaults to false.

Development

This package requires Node.js 18 or higher, pnpm, and a C++ compiler.

[!TIP] Enable pnpm log streaming to see full build output:

pnpm config set stream true

Building

There are two things being built: the native binding and the TypeScript code. Each of those can be built to be debug friendly.

Description	Command
Production build (minified + native binding)	pnpm build
TypeScript only (minified)	pnpm build:bundle
TypeScript only (unminified)	pnpm build:debug
Native binding only (prod)	pnpm rebuild
Native binding only (with debug logging)	pnpm rebuild:debug
Debug build everything	pnpm build:debug && pnpm rebuild:debug

When building the native binding, it will download the appropriate prebuilt RocksDB library for your platform and architecture from the rocksdb-prebuilds GitHub repository. It defaults to the pinned version in the package.json file. You can override this by setting the ROCKSDB_VERSION environment variable. For example:

ROCKSDB_VERSION=10.9.1 pnpm build

You may also specify latest to use the latest prebuilt version.

ROCKSDB_VERSION=latest pnpm build

Optionally, you may also create a .env file in the root of the project to specify various settings. For example:

echo "ROCKSDB_VERSION=10.9.1" >> .env

Linux C runtime versions

When you compile rocksdb-js, you can specify the ROCKSDB_LIBC environment variable to choose either glibc (default) or musl.

ROCKSDB_LIBC=musl pnpm rebuild

Windows C runtime versions

By default on Windows, rocksdb-js is compiled with the /MT flag. This will statically link the C runtime making the binary self-contained and portable.

Building RocksDB from Source

To build RocksDB from source, simply set the ROCKSDB_PATH environment variable to the path of the local rocksdb repo:

git clone https://github.com/facebook/rocksdb.git /path/to/rocksdb
echo "ROCKSDB_PATH=/path/to/rocksdb" >> .env
pnpm rebuild

Debugging

It is often helpful to do a debug build and see the internal debug logging of the native binding. You can do a debug build by running:

pnpm rebuild:debug

Each debug log message is prefixed with the thread id. Most debug log messages include the instance address making it easier to trace through the log output.

Debugging on macOS

In the event Node.js crashes, re-run Node.js in lldb:

lldb node
# Then in lldb:
# (lldb) run your-program.js
# When the crash occurs, print the stack trace:
# (lldb) bt

Testing

To run the tests, run:

pnpm coverage

To run the tests without code coverage, run:

pnpm test

To run a specific test suite, for example "ranges", run:

pnpm test ranges
# or
pnpm test test/ranges

To run a specific unit test, for example all tests that mention "column family", run:

pnpm test -t "column family"

Vitest's terminal renderer will often overwrite the debug log output, so it's highly recommended to specify the CI=1 environment variable to prevent Vitest from erasing log output:

CI=1 pnpm test

By default, the test runner deletes all test databases after the tests finish. To keep the temp databases for closer inspection, set the KEEP_FILES=1 environment variable:

CI=1 KEEP_FILES=1 pnpm test