DuckDB Node API
An API for using DuckDB in Node.
This is a high-level API meant for applications.
It depends on low-level bindings that adhere closely to DuckDB's C API,
available separately as @duckdb/node-bindings.
Features
Roadmap
Some features are not yet complete. See the issues list on GitHub
for the most up-to-date roadmap.
Supported Platforms
- Linux arm64 glibc
- Linux arm64 musl
- Linux x64 glibc
- Linux x64 musl
- Mac OS X (Darwin) arm64 (Apple Silicon)
- Mac OS X (Darwin) x64 (Intel)
- Windows (Win32) arm64
- Windows (Win32) x64
Examples
Get Basic Information
import duckdb from '@duckdb/node-api';
console.log(duckdb.version());
console.log(duckdb.configurationOptionDescriptions());
Connect
import { DuckDBConnection } from '@duckdb/node-api';
const connection = await DuckDBConnection.create();
This uses the default instance.
For advanced usage, you can create instances explicitly.
Create Instance
import { DuckDBInstance } from '@duckdb/node-api';
Create with an in-memory database:
const instance = await DuckDBInstance.create(':memory:');
Equivalent to the above:
const instance = await DuckDBInstance.create();
Read from and write to a database file, which is created if needed:
const instance = await DuckDBInstance.create('my_duckdb.db');
Set configuration options:
const instance = await DuckDBInstance.create('my_duckdb.db', {
threads: '4'
});
Instance Cache
Multiple instances in the same process should not
attach the same database.
To prevent this, an instance cache can be used:
const instance = await DuckDBInstance.fromCache('my_duckdb.db');
This uses the default instance cache. For advanced usage, you can create
instance caches explicitly:
import { DuckDBInstanceCache } from '@duckdb/node-api';
const cache = new DuckDBInstanceCache();
const instance = await cache.getOrCreateInstance('my_duckdb.db');
Connect to Instance
const connection = await instance.connect();
Disconnect
Connections will be disconnected automatically soon after their reference
is dropped, but you can also disconnect explicitly if and when you want:
connection.disconnectSync();
or, equivalently:
connection.closeSync();
Run SQL
const result = await connection.run('from test_all_types()');
Parameterize SQL
const prepared = await connection.prepare('select $1, $2, $3');
prepared.bindVarchar(1, 'duck');
prepared.bindInteger(2, 42);
prepared.bindList(3, listValue([10, 11, 12]), LIST(INTEGER));
const result = await prepared.run();
or:
const prepared = await connection.prepare('select $a, $b, $c');
prepared.bind({
'a': 'duck',
'b': 42,
'c': listValue([10, 11, 12]),
}, {
'a': VARCHAR,
'b': INTEGER,
'c': LIST(INTEGER),
});
const result = await prepared.run();
or even:
const result = await connection.run('select $a, $b, $c', {
'a': 'duck',
'b': 42,
'c': listValue([10, 11, 12]),
}, {
'a': VARCHAR,
'b': INTEGER,
'c': LIST(INTEGER),
});
Unspecified types will be inferred:
const result = await connection.run('select $a, $b, $c', {
'a': 'duck',
'b': 42,
'c': listValue([10, 11, 12]),
});
Specifying Values
Values of many data types are represented using one of the JS primitives
boolean, number, bigint, or string.
Also, any type can have null values.
Values of some data types need to be constructed using special functions.
These are:
ARRAY | arrayValue |
BIT | bitValue |
BLOB | blobValue |
DATE | dateValue |
DECIMAL | decimalValue |
INTERVAL | intervalValue |
LIST | listValue |
MAP | mapValue |
STRUCT | structValue |
TIME | timeValue |
TIMETZ | timeTZValue |
TIMESTAMP | timestampValue |
TIMESTAMPTZ | timestampTZValue |
TIMESTAMP_S | timestampSecondsValue |
TIMESTAMP_MS | timestampMillisValue |
TIMESTAMP_NS | timestampNanosValue |
UNION | unionValue |
UUID | uuidValue |
Stream Results
Streaming results evaluate lazily when rows are read.
const result = await connection.stream('from range(10_000)');
Inspect Result Metadata
Get column names and types:
const columnNames = result.columnNames();
const columnTypes = result.columnTypes();
Read Result Data
Run and read all data:
const reader = await connection.runAndReadAll('from test_all_types()');
const rows = reader.getRows();
Stream and read up to (at least) some number of rows:
const reader = await connection.streamAndReadUntil(
'from range(5000)',
1000
);
const rows = reader.getRows();
Read rows incrementally:
const reader = await connection.streamAndRead('from range(5000)');
reader.readUntil(2000);
reader.readUntil(4000);
reader.readUntil(6000);
Get Result Data
Result data can be retrieved in a variety of forms:
const reader = await connection.runAndReadAll(
'from range(3) select range::int as i, 10 + i as n'
);
const rows = reader.getRows();
const rowObjects = reader.getRowObjects();
const columns = reader.getColumns();
const columnsObject = reader.getColumnsObject();
Convert Result Data
By default, data values that cannot be represented as JS built-ins
are returned as specialized JS objects; see Inspect Data Values below.
To retrieve data in a different form, such as JS built-ins or values that
can be losslessly serialized to JSON, use the JS or Json forms of the
above result data methods.
Custom converters can be supplied as well. See the implementations of
JSDuckDBValueConverter
and JsonDuckDBValueConverters
for how to do this.
Examples (using the Json forms):
const reader = await connection.runAndReadAll(
'from test_all_types() select bigint, date, interval limit 2'
);
const rows = reader.getRowsJson();
const rowObjects = reader.getRowObjectsJson();
const columns = reader.getColumnsJson();
const columnsObject = reader.getColumnsObjectJson();
These methods handle nested types as well:
const reader = await connection.runAndReadAll(
'from test_all_types() select int_array, struct, map, "union" limit 2'
);
const rows = reader.getRowsJson();
const rowObjects = reader.getRowObjectsJson();
const columns = reader.getColumnsJson();
const columnsObject = reader.getColumnsObjectJson();
Column names and types can also be serialized to JSON:
const columnNamesAndTypes = reader.columnNamesAndTypesJson();
const columnNameAndTypeObjects = reader.columnNameAndTypeObjectsJson();
Fetch Chunks
Fetch all chunks:
const chunks = await result.fetchAllChunks();
Fetch one chunk at a time:
const chunks = [];
while (true) {
const chunk = await result.fetchChunk();
if (chunk.rowCount === 0) {
break;
}
chunks.push(chunk);
}
For materialized (non-streaming) results, chunks can be read by index:
const rowCount = result.rowCount;
const chunkCount = result.chunkCount;
for (let i = 0; i < chunkCount; i++) {
const chunk = result.getChunk(i);
}
Get chunk data:
const rows = chunk.getRows();
const rowObjects = chunk.getRowObjects(result.deduplicatedColumnNames());
const columns = chunk.getColumns();
const columnsObject =
chunk.getColumnsObject(result.deduplicatedColumnNames());
Get chunk data (one value at a time)
const columns = [];
const columnCount = chunk.columnCount;
for (let columnIndex = 0; columnIndex < columnCount; columnIndex++) {
const columnValues = [];
const columnVector = chunk.getColumnVector(columnIndex);
const itemCount = columnVector.itemCount;
for (let itemIndex = 0; itemIndex < itemCount; itemIndex++) {
const value = columnVector.getItem(itemIndex);
columnValues.push(value);
}
columns.push(columnValues);
}
Inspect Data Types
import { DuckDBTypeId } from '@duckdb/node-api';
if (columnType.typeId === DuckDBTypeId.ARRAY) {
const arrayValueType = columnType.valueType;
const arrayLength = columnType.length;
}
if (columnType.typeId === DuckDBTypeId.DECIMAL) {
const decimalWidth = columnType.width;
const decimalScale = columnType.scale;
}
if (columnType.typeId === DuckDBTypeId.ENUM) {
const enumValues = columnType.values;
}
if (columnType.typeId === DuckDBTypeId.LIST) {
const listValueType = columnType.valueType;
}
if (columnType.typeId === DuckDBTypeId.MAP) {
const mapKeyType = columnType.keyType;
const mapValueType = columnType.valueType;
}
if (columnType.typeId === DuckDBTypeId.STRUCT) {
const structEntryNames = columnType.names;
const structEntryTypes = columnType.valueTypes;
}
if (columnType.typeId === DuckDBTypeId.UNION) {
const unionMemberTags = columnType.memberTags;
const unionMemberTypes = columnType.memberTypes;
}
if (columnType.alias === 'JSON') {
const json = JSON.parse(columnValue);
}
Every type implements toString.
The result is both human-friendly and readable by DuckDB in an appropriate expression.
const typeString = columnType.toString();
Inspect Data Values
import { DuckDBTypeId } from '@duckdb/node-api';
if (columnType.typeId === DuckDBTypeId.ARRAY) {
const arrayItems = columnValue.items;
const arrayString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.BIT) {
const bools = columnValue.toBools();
const bits = columnValue.toBits();
const bitString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.BLOB) {
const blobBytes = columnValue.bytes;
const blobString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.DATE) {
const dateDays = columnValue.days;
const dateString = columnValue.toString();
const { year, month, day } = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.DECIMAL) {
const decimalWidth = columnValue.width;
const decimalScale = columnValue.scale;
const decimalValue = columnValue.value;
const decimalString = columnValue.toString();
const decimalDouble = columnValue.toDouble();
}
if (columnType.typeId === DuckDBTypeId.INTERVAL) {
const intervalMonths = columnValue.months;
const intervalDays = columnValue.days;
const intervalMicros = columnValue.micros;
const intervalString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.LIST) {
const listItems = columnValue.items;
const listString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.MAP) {
const mapEntries = columnValue.entries;
const mapString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.STRUCT) {
const structEntries = columnValue.entries;
const structString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_MS) {
const timestampMillis = columnValue.milliseconds;
const timestampMillisString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_NS) {
const timestampNanos = columnValue.nanoseconds;
const timestampNanosString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_S) {
const timestampSecs = columnValue.seconds;
const timestampSecsString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_TZ) {
const timestampTZMicros = columnValue.micros;
const timestampTZString = columnValue.toString();
const {
date: { year, month, day },
time: { hour, min, sec, micros },
} = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP) {
const timestampMicros = columnValue.micros;
const timestampString = columnValue.toString();
const {
date: { year, month, day },
time: { hour, min, sec, micros },
} = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.TIME_TZ) {
const timeTZMicros = columnValue.micros;
const timeTZOffset = columnValue.offset;
const timeTZString = columnValue.toString();
const {
time: { hour, min, sec, micros },
offset,
} = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.TIME) {
const timeMicros = columnValue.micros;
const timeString = columnValue.toString();
const { hour, min, sec, micros } = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.UNION) {
const unionTag = columnValue.tag;
const unionValue = columnValue.value;
const unionValueString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.UUID) {
const uuidHugeint = columnValue.hugeint;
const uuidString = columnValue.toString();
}
Displaying Timezones
Converting a TIMESTAMP_TZ value to a string depends on a timezone offset.
By default, this is set to the offset for the local timezone when the Node
process is started.
To change it, set the timezoneOffsetInMinutes
property of DuckDBTimestampTZValue:
DuckDBTimestampTZValue.timezoneOffsetInMinutes = -8 * 60;
const pst = DuckDBTimestampTZValue.Epoch.toString();
DuckDBTimestampTZValue.timezoneOffsetInMinutes = +1 * 60;
const cet = DuckDBTimestampTZValue.Epoch.toString();
Note that the timezone offset used for this string
conversion is distinct from the TimeZone setting of DuckDB.
The following sets this offset to match the TimeZone setting of DuckDB:
const reader = await connection.runAndReadAll(
`select (timezone(current_timestamp) / 60)::int`
);
DuckDBTimestampTZValue.timezoneOffsetInMinutes =
reader.getColumns()[0][0];
Append To Table
await connection.run(
`create or replace table target_table(i integer, v varchar)`
);
const appender = await connection.createAppender('target_table');
appender.appendInteger(42);
appender.appendVarchar('duck');
appender.endRow();
appender.appendInteger(123);
appender.appendVarchar('mallard');
appender.endRow();
appender.flushSync();
appender.appendInteger(17);
appender.appendVarchar('goose');
appender.endRow();
appender.closeSync();
Append Data Chunk
await connection.run(
`create or replace table target_table(i integer, v varchar)`
);
const appender = await connection.createAppender('target_table');
const chunk = DuckDBDataChunk.create([INTEGER, VARCHAR]);
chunk.setColumns([
[42, 123, 17],
['duck', 'mallad', 'goose'],
]);
appender.appendDataChunk(chunk);
appender.flushSync();
See "Specifying Values" above for how to supply values to the appender.
Scalar Functions
connection.registerScalarFunction(
DuckDBScalarFunction.create({
name: 'my_add',
mainFunction: (info, input, output) => {
const v0 = input.getColumnVector(0);
const v1 = input.getColumnVector(1);
for (let rowIndex = 0; rowIndex < input.rowCount; rowIndex++) {
output.setItem(
rowIndex,
v0.getItem(rowIndex) + v1.getItem(rowIndex)
);
}
output.flush();
},
returnType: INTEGER,
parameterTypes: [INTEGER, INTEGER],
})
);
const reader = await connection.runAndReadAll('select my_add(2, 3)');
const rows = reader.getRows();
const extractedStatements = await connection.extractStatements(`
create or replace table numbers as from range(?);
from numbers where range < ?;
drop table numbers;
`);
const parameterValues = [10, 7];
const statementCount = extractedStatements.count;
for (let stmtIndex = 0; stmtIndex < statementCount; stmtIndex++) {
const prepared = await extractedStatements.prepare(stmtIndex);
let parameterCount = prepared.parameterCount;
for (let paramIndex = 1; paramIndex <= parameterCount; paramIndex++) {
prepared.bindInteger(paramIndex, parameterValues.shift());
}
const result = await prepared.run();
}
Control Evaluation of Tasks
DuckDB splits work into relatively short tasks. By controlling the
evaluation of these tasks explicitly, better cooperative multithreading
can be accomplished. This is especially important for maximum throughput
in the Node environment, which has a small, fixed number of threads for
running asynchronous work.
(See https://docs.libuv.org/en/stable/threadpool.html)
Below is a low-level way of controlling the evaluation of processing.
See the "startThenRead" and "startStreamThenRead" methods for helpers
that do much of this for you.
import { DuckDBPendingResultState } from '@duckdb/node-api';
async function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
const prepared = await connection.prepare('from range(10_000_000)');
const pending = prepared.startStream();
while (pending.runTask() !== DuckDBPendingResultState.RESULT_READY) {
console.log('not ready');
await sleep(1);
}
console.log('ready');
const result = await pending.getResult();
Equivalently, using startStreamThenRead:
const reader = await connection.startStreamThenRead(sql);
Ways to run SQL
const result = await connection.run(sql);
const result = await connection.run(sql, values);
const result = await connection.run(sql, values, types);
const reader = await connection.runAndRead(sql);
const reader = await connection.runAndRead(sql, values);
const reader = await connection.runAndRead(sql, values, types);
const reader = await connection.runAndReadAll(sql);
const reader = await connection.runAndReadAll(sql, values);
const reader = await connection.runAndReadAll(sql, values, types);
const reader = await connection.runAndReadUntil(sql, targetRowCount);
const reader =
await connection.runAndReadAll(sql, targetRowCount, values);
const reader =
await connection.runAndReadAll(sql, targetRowCount, values, types);
const result = await connection.stream(sql);
const result = await connection.stream(sql, values);
const result = await connection.stream(sql, values, types);
const reader = await connection.streamAndRead(sql);
const reader = await connection.streamAndRead(sql, values);
const reader = await connection.streamAndRead(sql, values, types);
const reader = await connection.streamAndReadAll(sql);
const reader = await connection.streamAndReadAll(sql, values);
const reader = await connection.streamAndReadAll(sql, values, types);
const reader = await connection.streamAndReadUntil(sql, targetRowCount);
const reader =
await connection.streamAndReadUntil(sql, targetRowCount, values);
const reader =
await connection.streamAndReadUntil(sql, targetRowCount, values, types);
const prepared = await connection.prepare(sql);
prepared.bind(values);
prepared.bind(values, types);
const result = prepared.run();
const reader = prepared.runAndRead();
const reader = prepared.runAndReadAll();
const reader = prepared.runAndReadUntil(targetRowCount);
const result = prepared.stream();
const reader = prepared.streamAndRead();
const reader = prepared.streamAndReadAll();
const reader = prepared.streamAndReadUntil(targetRowCount);
const pending = await connection.start(sql);
const pending = await connection.start(sql, values);
const pending = await connection.start(sql, values, types);
const reader = await connection.startThenRead(sql);
const reader = await connection.startThenRead(sql, values);
const reader = await connection.startThenRead(sql, values, types);
const reader = await connection.startThenReadAll(sql);
const reader = await connection.startThenReadAll(sql, values);
const reader = await connection.startThenReadAll(sql, values, types);
const reader = await connection.startThenReadUntil(sql, targetRowCount);
const reader =
await connection.startThenReadUntil(sql, targetRowCount, values);
const reader =
await connection.startThenReadUntil(sql, targetRowCount, values, types);
const pending = await connection.startStream(sql);
const pending = await connection.startStream(sql, values);
const pending = await connection.startStream(sql, values, types);
const reader = await connection.startStreamThenRead(sql);
const reader = await connection.startStreamThenRead(sql, values);
const reader = await connection.startStreamThenRead(sql, values, types);
const reader = await connection.startStreamThenReadAll(sql);
const reader = await connection.startStreamThenReadAll(sql, values);
const reader =
await connection.startStreamThenReadAll(sql, values, types);
const reader =
await connection.startStreamThenReadUntil(sql, targetRowCount);
const reader =
await connection.startStreamThenReadUntil(sql, targetRowCount, values);
const reader = await connection.startStreamThenReadUntil(
sql, targetRowCount, values, types);
const pending = await prepared.start();
const pending = await prepared.startStream();
while (pending.runTask() !== DuckDBPendingResultState.RESULT_READY) {
}
await pending.runAllTasks();
const result = await pending.getResult();
const reader = await pending.read();
const reader = await pending.readAll();
const reader = await pending.readUntil(targetRowCount);
Ways to get result data
const columns = await result.getColumns();
const columnsJS = await result.getColumnsJS();
const columnsJson = await result.getColumnsJson();
const columnsObject = await result.getColumnsObject();
const columnsObjectJS = await result.getColumnsObjectJS();
const columnsObjectJson = await result.getColumnsObjectJson();
const rows = await result.getRows();
const rowsJS = await result.getRowsJS();
const rowsJson = await result.getRowsJson();
const rowObjects = await result.getRowObjects();
const rowObjectsJS = await result.getRowObjectsJS();
const rowObjectsJson = await result.getRowObjectsJson();
for await (const chunk of result) {
}
for await (const rows of result.yieldRows()) {
}
await reader.readAll();
await reader.readUntil(targetRowCount);
const columns = reader.getColumns();
const columnsJS = reader.getColumnsJS();
const columnsJson = reader.getColumnsJson();
const columnsObject = reader.getColumnsObject();
const columnsObjectJS = reader.getColumnsObjectJS();
const columnsObjectJson = reader.getColumnsObjectJson();
const rows = reader.getRows();
const rowsJS = reader.getRowsJS();
const rowsJson = reader.getRowsJson();
const rowObjects = reader.getRowObjects();
const rowObjectsJS = reader.getRowObjectsJS();
const rowObjectsJson = reader.getRowObjectsJson();
const value = reader.value(columnIndex, rowIndex);
const chunk = await result.fetchChunk();
const chunks = await result.fetchAllChunks();
const columnValues = chunk.getColumnValues(columnIndex);
const columns = chunk.getColumns();
const rowValues = chunk.getRowValues(rowIndex);
const rows = chunk.getRows();
chunk.visitColumnValues(columnIndex,
(value, rowIndex, columnIndex, type) => { }
);
chunk.visitColumns((column, columnIndex, type) => { });
chunk.visitColumnMajor(
(value, rowIndex, columnIndex, type) => { }
);
chunk.visitRowValues(rowIndex,
(value, rowIndex, columnIndex, type) => { }
);
chunk.visitRows((row, rowIndex) => { });
chunk.visitRowMajor(
(value, rowIndex, columnIndex, type) => { }
);
const columnValues = chunk.convertColumnValues(columnIndex, converter);
const columns = chunk.convertColumns(converter);
const rowValues = chunk.convertRowValues(rowIndex, converter);
const rows = chunk.convertRows(converter);