@dikolab/vdb

@dikolab/vdb

A multi-partition vector database: lexical (BM25F), vector (cosine), and hybrid (RRF) search over named partitions — each with its own indexes, field weights, and embedding provider — searchable one partition at a time or across a dynamic slice. Beyond ranked search it is also a simple partitioned record store: list + CRUD where a stable-id record is attached to a many-to-many selection of partitions. Runs the same on Node.js and Deno.

It is a thin TypeScript wrapper over a Rust core compiled to WebAssembly with wasm-pack — the WASM does the scoring and embedding, and the TypeScript owns the store layout, indexes, CLI, and daemon. The WASM loads locally on Node and offline on Deno (it ships in the module graph, so deno cache / deno install cover it — no per-process network fetch).

Documentation

Full documentation is published on the project docs site:

• Overview — features, install, and a quick start.
• API Reference — the complete Vdb client surface, standalone primitives, and enums.
• Examples — end-to-end store, partition, record, batch-write/delete, search, and maintenance usage.
• Releases — every published version and what it added.

A comprehensive reference ships in this package under docs/: partitions, records, search, storage, configuration, architecture (the Rust → WASM crates), CLI, API, integration, and embedding vdb.

Install

npm add @dikolab/vdb          # npm / Node.js / bundlers
deno add jsr:@dikolab/vdb     # Deno (JSR)

The package ships as ES modules only.

Requires Deno ≥ 2.6.0 (for source-phase WebAssembly imports, which let the WASM load offline). Node is unaffected — engines.node: >=18 is unchanged.

Quick start

Open a client against a store, declare a partition with a column schema, create records, and search:

import {
   createClient,
   ExecutionMode,
   AttributeType,
   SearchAlgo,
} from "@dikolab/vdb";

const vdb = createClient({ db: "./store", mode: ExecutionMode.InProcess });

// Declare a partition and the columns records may carry (full-text columns are searchable).
await vdb.createPartition("docs", {
   title: { type: AttributeType.String, fullTextSearch: true, weight: 2 },
   body: { type: AttributeType.String, fullTextSearch: true, weight: 1 },
});

// Create a record attached to one or more partitions.
const rec = await vdb.create({
   partitions: ["docs"],
   attributes: [
      { field: "title", value: "Install Guide" },
      { field: "body", value: "Run npm add @dikolab/vdb to get started." },
   ],
});

// Ingesting many at once? `createMany` reindexes the batch ONCE (O(N), not the
// O(N²) of N eager creates) and reports a per-record outcome. (Follow-up per-record
// updates reindex again — defer them with `{ reindex: false }` and end the batch
// with one `rebuild()`. A deferred, un-settled write is detectable:
// `status().pendingRebuild` is true and `check()` fails with a `stale-index` error.)
await vdb.createMany([
   { partitions: ["docs"], attributes: [{ field: "body", value: "first" }] },
   { partitions: ["docs"], attributes: [{ field: "body", value: "second" }] },
]);

// Removing many is the same shape: `deleteMany` settles ONCE across every partition
// the batch touched. A missing id is not an error (`ok: true`, absent from
// `deleted`), and input order does not matter — deletes never cascade.
await vdb.deleteMany(["doc", "member-1", "already-gone"]);

// Unranked list, filtered by partition.
await vdb.list({ partitions: ["docs"], limit: 20 });

// Ranked search — each partition ranks with its own config; a slice fuses.
const hits = await vdb.search({
   query: "install",
   partitions: ["docs"],
   algo: SearchAlgo.Hybrid,
});
for (const hit of hits.items) {
   console.log(hit.score.toFixed(3), hit.matchedTerms, hit.snippet);
}

Or from the CLI:

vdb db init --db ./store
vdb partition create docs --db ./store --column 'title:string:fulltext:2' --column 'body:string:fulltext:1'
vdb record create --db ./store --partition docs --attr 'title=Install Guide' --attr 'body=Run the installer.'
vdb search "install" --db ./store --partition docs --algo hybrid

Search capabilities

• Lexical, vector & hybrid — BM25F full-text, cosine vector, and RRF-hybrid search per partition, with rank-fused cross-partition slices.
• Approximate vector search (ANN) — opt in per partition to an HNSW index (ann.enabled) that generates candidates sub-linearly and then re-scores them exactly by cosine, so ranking is unchanged; off by default, with the exact scan the default and the fallback for small partitions.
• Query grammar — a quoted "exact phrase" (positional), -term negation, and boolean AND / OR / (…); a plain bag-of-words query is unchanged and fully backward-compatible.
• Recall expansion — curated synonyms (login⇄signin, config⇄configuration, …) and adjacent-word n-grams widen recall without perturbing base ranking; on by default, toggleable per partition (synonyms/ngrams, or --synonyms/--ngrams).
• Relevance signals — each hit carries a top-relative confidence in [0, 1] beside the raw score, plus matchedTerms and matchedFields.
• Deterministic ranking — a pure function of (corpus, query): exactly-tied scores break by ascending id, so ordering is reproducible across rebuilds, insertion orders, and runtimes.
• Engine metrics, readiness & integrity — stats()/status() expose corpus, index, and daemon-cache figures (distinctTerms, termsIndexed, cacheStats, wasmMemoryBytes); status() is cheap (a record file count, with no embedding module loaded for a provider: "none" store); ready() is a cheap probe that forces the engine to load; hasPendingRebuild() is a WASM-free stat for a fast owed-rebuild poll; check() validates the store and surfaces best-effort semanticContradictions.
• Fusion primitive — fuseRankedSets fuses N externally-ranked sets (one primary) into a single deduped, ranked, paginated page — the engine's rank fusion, exported for reuse.
• Diversity & rerank — opt-in SearchParams.diversity (MMR) cuts near-duplicate crowding using record vectors (exported as mmrSelect); opt-in SearchParams.rerank reorders the top with an injected RerankProvider (Node-only onnxCrossEncoder, or bring your own). Both are strict no-ops when absent.

Store capabilities

Beyond ranked search, vdb is a partitioned record store — the unranked half of the same engine.

• Records & partitions — a stable-id record carries typed, schema-declared attributes and is attached to a many-to-many set of named partitions (attach/detach adjust membership without rewriting the record). Reserved fields (checksum, size, createdAt, updatedAt) are engine-managed.
• List (multi-valued) columns — declare list: true and a column holds a set, filtered by set-membership rather than equality (tags, a document's members).
• Edges & the reference graph — caller-supplied outbound edges ({ to, type?, context? }) with outboundEdges/inboundEdges queries over a derived, bidirectional index. compose(ids) expands a document into its ordered, deduped, cycle-safe member records.
• Batch writes and batch deletes — createMany(inputs) and deleteMany(ids) apply a whole batch and reindex the touched partitions once: O(N) instead of the O(N²) of N eager create/delete calls, with no index-stale window and a per-item outcome. deleteMany treats a missing id as an ordinary outcome (not an error), is order-independent (deletes never cascade), and settles every partition the batch touched. Alternatively { reindex: false } defers the index to one rebuild().
• Cheap lookups — getBy({ partition, field, value }) resolves attribute equality off the in-memory metadata cache (O(partition) in memory, not the O(store) scan list pays), and count(params) returns a total without materialising rows. Both carry a settled flag distinguishing "genuinely empty" from "index not built here"; list stays a pool scan on purpose, for settle-immediate reads that gate a write.
• Bulk reads — getMany(ids) (O(ids) single-file reads), readVectors({ partition, ids? }) for stored embeddings, pairsAbove({ minScore, ids? }) for every record pair above a cosine threshold in a WASM kernel, and searchMany(queries) for one page per query.
• Detectable staleness — a deferred write arms a durable pending-rebuild marker, so a crash mid-batch is never silently stale: status().pendingRebuild, the WASM-free hasPendingRebuild(), and a check() stale-index error all surface it until a rebuild() clears it.
• Opt-in durability — open the store with durable: true (or --durable) for fsync-backed, atomic tmp+rename writes that survive machine death, not just process death. It is a write-path option, not a format: a default client reads and writes the same store unchanged. Off by default; see Durability.
• Maintenance — rebuild(), check() (integrity + best-effort semantic contradictions), gc() (reclaims orphaned vector and edge index rows), stats(), exportStore()/importStore(), and migrate() to adopt a legacy version-2 store (optionally collapsing it into one named partition).

Full API surface

Every public member of the Vdb client, grouped. Full signatures and types: API.

Group	Members
Lifecycle	connect · init · ready · close · startDaemon · stopDaemon · db · mode
Records (read)	get · getMany · list · getBy · count · compose
Records (write)	create · createMany · update · delete · deleteMany · attach · detach
Edges	outboundEdges · inboundEdges
Search	search · searchMany · recall · content
Vectors	readVectors · pairsAbove
Partitions	createPartition · getPartition · updatePartition · deletePartition · listPartitions
Maintenance	rebuild · hasPendingRebuild · status · stats · check · gc · migrate · exportStore · importStore

Standalone exports: createClient · contentId · attributeValues · fuseRankedSets · mmrSelect · onnxCrossEncoder · DB_FORMAT_VERSION, plus the ExecutionMode, AttributeType, SearchAlgo, SearchMode and BoostRole enums and every result/param type.

Three entrypoints, one engine

• @dikolab/vdb — the library (createClient → Vdb); what a host application imports.
• @dikolab/vdb/cli — the thin vdb CLI (daemon, partitions, records, search).
• @dikolab/vdb/worker — the detached daemon a daemon-mode client spawns.

Run the engine in-process (embed it, no IPC) or via a shared per-store daemon — see Embedding vdb.

Content-addressed ids

A record's id is a ULID by default, or you may bring your own — pass create({ id, … }) to store under a caller-supplied id, and re-creating the same id upserts in place (preserving createdAt). The exported contentId(text) helper turns any text into a stable, portable content hash you can use as that id, so identical content maps to the same record.

Support

If @dikolab/vdb is useful to you, you can support its development:

Support development

License

AGPL-3.0-only © 2026 Diko Consunji