@dikolab/vdb

@dikolab/vdb

A multi-partition vector database: lexical (BM25F), vector (cosine), and hybrid (RRF) search over named partitions — each with its own indexes, field weights, and embedding provider — searchable one partition at a time or across a dynamic slice. Beyond ranked search it is also a simple partitioned record store: list + CRUD where a stable-id record is attached to a many-to-many selection of partitions. Runs the same on Node.js and Deno.

It is a thin TypeScript wrapper over a Rust core compiled to WebAssembly with wasm-pack — the WASM does the scoring and embedding, and the TypeScript owns the store layout, indexes, CLI, and daemon. The WASM loads locally on Node and offline on Deno (it ships in the module graph, so deno cache / deno install cover it — no per-process network fetch).

Documentation

Full documentation is published on the project docs site:

• Overview — features, install, and a quick start.
• Examples — end-to-end store, partition, record, and search usage.

A comprehensive reference ships in this package under docs/: partitions, records, search, storage, configuration, architecture (the Rust → WASM crates), CLI, API, integration, and embedding vdb.

Install

npm add @dikolab/vdb          # npm / Node.js / bundlers
deno add jsr:@dikolab/vdb     # Deno (JSR)

The package ships as ES modules only.

Requires Deno ≥ 2.6.0 (for source-phase WebAssembly imports, which let the WASM load offline). Node is unaffected — engines.node: >=18 is unchanged.

Quick start

Open a client against a store, declare a partition with a column schema, create records, and search:

import {
   createClient,
   ExecutionMode,
   AttributeType,
   SearchAlgo,
} from "@dikolab/vdb";

const vdb = createClient({ db: "./store", mode: ExecutionMode.InProcess });

// Declare a partition and the columns records may carry (full-text columns are searchable).
await vdb.createPartition("docs", {
   title: { type: AttributeType.String, fullTextSearch: true, weight: 2 },
   body: { type: AttributeType.String, fullTextSearch: true, weight: 1 },
});

// Create a record attached to one or more partitions.
const rec = await vdb.create({
   partitions: ["docs"],
   attributes: [
      { field: "title", value: "Install Guide" },
      { field: "body", value: "Run npm add @dikolab/vdb to get started." },
   ],
});

// Ingesting many at once? `createMany` reindexes the batch ONCE (O(N), not the
// O(N²) of N eager creates) and reports a per-record outcome.
await vdb.createMany([
   { partitions: ["docs"], attributes: [{ field: "body", value: "first" }] },
   { partitions: ["docs"], attributes: [{ field: "body", value: "second" }] },
]);

// Unranked list, filtered by partition.
await vdb.list({ partitions: ["docs"], limit: 20 });

// Ranked search — each partition ranks with its own config; a slice fuses.
const hits = await vdb.search({
   query: "install",
   partitions: ["docs"],
   algo: SearchAlgo.Hybrid,
});
for (const hit of hits.items) {
   console.log(hit.score.toFixed(3), hit.matchedTerms, hit.snippet);
}

Or from the CLI:

vdb db init --db ./store
vdb partition create docs --db ./store --column 'title:string:fulltext:2' --column 'body:string:fulltext:1'
vdb record create --db ./store --partition docs --attr 'title=Install Guide' --attr 'body=Run the installer.'
vdb search "install" --db ./store --partition docs --algo hybrid

Search capabilities

• Lexical, vector & hybrid — BM25F full-text, cosine vector, and RRF-hybrid search per partition, with rank-fused cross-partition slices.
• Query grammar — a quoted "exact phrase" (positional), -term negation, and boolean AND / OR / (…); a plain bag-of-words query is unchanged and fully backward-compatible.
• Recall expansion — curated synonyms (login⇄signin, config⇄configuration, …) and adjacent-word n-grams widen recall without perturbing base ranking; on by default, toggleable per partition (synonyms/ngrams, or --synonyms/--ngrams).
• Relevance signals — each hit carries a top-relative confidence in [0, 1] beside the raw score, plus matchedTerms and matchedFields.
• Deterministic ranking — a pure function of (corpus, query): exactly-tied scores break by ascending id, so ordering is reproducible across rebuilds, insertion orders, and runtimes.
• Engine metrics, readiness & integrity — stats()/status() expose corpus, index, and daemon-cache figures (distinctTerms, termsIndexed, cacheStats, wasmMemoryBytes); status() is cheap (a record file count, with no embedding module loaded for a provider: "none" store); ready() is a cheap probe that forces the engine to load; check() validates the store and surfaces best-effort semanticContradictions.
• Fusion primitive — fuseRankedSets fuses N externally-ranked sets (one primary) into a single deduped, ranked, paginated page — the engine's rank fusion, exported for reuse.
• Diversity & rerank — opt-in SearchParams.diversity (MMR) cuts near-duplicate crowding using record vectors (exported as mmrSelect); opt-in SearchParams.rerank reorders the top with an injected RerankProvider (Node-only onnxCrossEncoder, or bring your own). Both are strict no-ops when absent.

Three entrypoints, one engine

• @dikolab/vdb — the library (createClient → Vdb); what a host application imports.
• @dikolab/vdb/cli — the thin vdb CLI (daemon, partitions, records, search).
• @dikolab/vdb/worker — the detached daemon a daemon-mode client spawns.

Run the engine in-process (embed it, no IPC) or via a shared per-store daemon — see Embedding vdb.

Content-addressed ids

A record's id is a ULID by default, or you may bring your own — pass create({ id, … }) to store under a caller-supplied id, and re-creating the same id upserts in place (preserving createdAt). The exported contentId(text) helper turns any text into a stable, portable content hash you can use as that id, so identical content maps to the same record.

Support

If @dikolab/vdb is useful to you, you can support its development:

Support development

License

AGPL-3.0-only © 2026 Diko Consunji