tokenu

tokenu

A Unix du-like CLI that counts token usage per file and directory.

Quick Start

No install needed — run it directly with npx:

npx tokenu .

Or install it globally:

npm install -g tokenu

Usage

tokenu [options] [path...]

Options

Flag	Description
-s, --summarize	Display only a total for each argument
-h, --human-readable	Print token counts in human-readable format (1K, 1M)
-a, --all	Show counts for all files, not just directories
-d, --max-depth <N>	Print totals only for directories N levels deep
-c, --total	Produce a grand total
--json	Output as JSON (for AI agent consumption)
--encoding <enc>	Tokenizer encoding (default: o200k_base)
--model <name>	Model name (e.g. gpt-4o, gpt-3.5-turbo)
--exclude <pat>	Glob pattern to exclude (repeatable)

Supported encodings: o200k_base, o200k_harmony, cl100k_base, p50k_base, p50k_edit, r50k_base

Examples

Recursive token counts per directory:

$ tokenu -a src/
1127	src/bin/cli.ts
1127	src/bin
783	src/formatter.ts
256	src/main.ts
324	src/tokenizer.ts
165	src/types.ts
735	src/walker.ts
3390	src

Human-readable summary:

$ tokenu -hs src/
3.4K	src

Depth-limited with grand total:

$ tokenu -d 1 --total myproject/
4143	myproject/tests
2019	myproject/docs
2263	myproject/src
12241	myproject/dist
20666	myproject
20666	total

JSON output (useful for piping to other tools or AI agents):

$ tokenu -a --json src/

Use a specific encoding for older models:

$ tokenu --encoding cl100k_base .

Contributing

Please consult CONTRIBUTING for guidelines on contributing to this project.

Author

tokenu © Liran Tal, Released under the Apache-2.0 License.