@winci/bun-chunk

bun-chunk

AST-aware code chunking for RAG pipelines — fast, simple, Bun-native.

Uses tree-sitter to parse source code into an AST, then splits it into semantically meaningful chunks (functions, classes, interfaces, etc.) instead of naive line-based splitting. This produces better embeddings and more relevant retrieval for code search and RAG.

Supported languages

TypeScript, JavaScript, Python, Rust, Go, Java, C, C++, C#, Ruby, PHP, Scala, HTML, CSS, Kotlin, Lua, Zig, Elixir, Bash, TOML, YAML, Haskell, OCaml, Dart

Future candidates

These languages need a compatible tree-sitter WASM binary before they can be added:

• Swift — no npm package ships a .wasm; requires self-compiling or the tree-sitter-wasms mega-bundle
• SQL — no npm package ships a .wasm; best grammar is @derekstride/tree-sitter-sql but needs manual WASM build

Install

bun add @winci/bun-chunk

Usage

import { chunk } from "@winci/bun-chunk";

const code = await Bun.file("src/app.ts").text();
const chunks = await chunk("src/app.ts", code);

for (const c of chunks) {
  console.log(`[${c.type}] ${c.name ?? "(anonymous)"} (lines ${c.startLine}-${c.endLine})`);
  console.log(c.text);
}

Options

const chunks = await chunk("src/app.ts", code, {
  maxLines: 40,                // max chunk size in lines (default: 60)
  language: "typescript",      // override language detection
  strategy: "semantic",        // "semantic" (default) | "fixed" | "hybrid"
  overlap: 3,                  // lines of overlap between chunks (default: 0)
  includeMetadata: true,       // add language, filePath, hash to each chunk
  includeContext: true,        // add parent scope context to child chunks
});

Language is auto-detected from the file extension. Supported extensions: .ts, .tsx, .mts, .cts, .js, .jsx, .mjs, .cjs, .py, .pyi, .rs, .go, .java, .c, .h, .cpp, .cc, .cxx, .hpp, .hh, .hxx, .cs, .rb, .php, .scala, .sc, .html, .htm, .css, .scss, .less, .kt, .kts, .lua, .zig, .zon, .ex, .exs, .sh, .bash, .zsh, .toml, .yaml, .yml, .hs, .lhs, .ml, .mli, .dart.

Strategies:

• semantic (default) — AST-aware, extracts named entities
• fixed — Pure line-based splitting, no AST
• hybrid — AST for known languages, fallback to fixed for unknowns

Merging small chunks

Adjacent small chunks (blocks, imports) can be merged to reduce total chunk count:

import { chunk, mergeSmallChunks } from "@winci/bun-chunk";

const chunks = await chunk("app.ts", code);
const merged = mergeSmallChunks(chunks, 60);

File and directory chunking

Chunk a single file from disk:

import { chunkFile } from "@winci/bun-chunk";

const result = await chunkFile("src/app.ts", { maxLines: 40 });
// result.chunks, result.fileImports, result.fileExports

Chunk an entire directory:

import { chunkDirectory } from "@winci/bun-chunk";

const result = await chunkDirectory("src/", {
  glob: "**/*.ts",
  ignore: ["node_modules", "dist"],
  onProgress({ filePath, chunkCount, fileIndex, totalFiles }) {
    console.log(`[${fileIndex}/${totalFiles}] ${filePath}: ${chunkCount} chunks`);
  },
});

for (const [filePath, fileResult] of result.files) {
  console.log(filePath, fileResult.chunks.length);
}

Streaming

All chunking functions have streaming counterparts that yield chunks one at a time:

import { chunkStream, chunkFileStream, chunkDirectoryStream } from "@winci/bun-chunk";

for await (const chunk of chunkFileStream("src/app.ts")) {
  console.log(chunk.name, chunk.type);
}

Structured imports and exports

Every chunk result includes parsed imports and exports:

import { chunk } from "@winci/bun-chunk";

const result = await chunk("src/app.ts", code);

for (const imp of result.fileImports) {
  console.log(imp.name, imp.source, imp.isDefault, imp.isNamespace);
}

for (const exp of result.fileExports) {
  console.log(exp.name, exp.type, exp.isDefault, exp.isReExport);
}

You can also extract them directly:

import { extractImports, extractExports } from "@winci/bun-chunk";

Project-level analysis

chunkProject resolves cross-file imports and builds a dependency graph:

import { chunkProject } from "@winci/bun-chunk";

const project = await chunkProject("src/", {
  glob: "**/*.ts",
  resolveImports: true, // default
});

// Dependency graph
for (const [file, deps] of project.graph.importsFrom) {
  console.log(`${file} imports from:`, [...deps]);
}

for (const [file, dependents] of project.graph.importedBy) {
  console.log(`${file} is imported by:`, [...dependents]);
}

// Chunks are enriched with cross-file info
for (const [filePath, ctx] of project.files) {
  for (const chunk of ctx.result.chunks) {
    if (chunk.usedBy) console.log(`${chunk.name} used by:`, chunk.usedBy);
    if (chunk.definedIn) console.log(`${chunk.name} defined in:`, chunk.definedIn);
  }
}

CLI

bunx @winci/bun-chunk <file|directory> [options]

Option	Description
-f, --format <json|jsonl|text>	Output format (default: json)
-m, --max-lines <n>	Max chunk size in lines (default: 60)
-l, --language <lang>	Override language detection
-s, --strategy <semantic|fixed|hybrid>	Chunking strategy (default: semantic)
--overlap <n>	Line overlap between chunks (default: 0)
-g, --glob <pattern>	File filter for directory mode
--include-metadata	Include language, filePath, hash
--include-context	Include parent scope context

How it works

• Parses source code into an AST using tree-sitter (WASM grammars)
• Runs language-specific queries to extract top-level entities (functions, classes, types, imports, etc.)
• Consecutive imports are collapsed into single chunks
• Leading comments and decorators attach to their entity
• Creates one chunk per entity that fits within maxLines
• Splits oversized entities by their children (e.g., methods within a class)
• Falls back to line-based splitting for very large blocks or unsupported languages

Each chunk includes:

Field	Type	Description
text	string	The chunk's source code
startLine	number	0-indexed start line
endLine	number	0-indexed end line (inclusive)
type	ChunkType	Entity type (function, class, interface, block, etc.)
name	string | null	Entity name if available
imports	ChunkImport[]	Imports in this chunk
exports	ChunkExport[]	Exports in this chunk
context	string[]	Parent scope chain (with includeContext)
parentName	string	Enclosing entity name
language	string	Detected language (with includeMetadata)
filePath	string	Source file path (with includeMetadata)
hash	string	Content SHA256 hash (with includeMetadata)
usedBy	string[]	Files importing this export (via chunkProject)
definedIn	string[]	Files defining this import (via chunkProject)

License

MIT