Sign In

tokenu

Package Overview
Dependencies
Maintainers
1
Versions
3
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

tokenu - npm Package Compare versions

Comparing version
1.0.0
 to
1.0.1
+14
-15
package.json
{
"name": "tokenu",
"version": "1.0.0",
"version": "1.0.1",
"description": "A Unix du-like command line tool to count token usage per files and directories",

@@ -31,3 +31,2 @@ "types": "dist/main.d.mts",

},
"packageManager": "pnpm@10.31.0",
"files": [

@@ -38,15 +37,2 @@ "dist",

],
"scripts": {
"start": "tsx src/bin/cli.ts",
"build": "tsc && tsdown",
"lint": "eslint . && pnpm run lint:markdown",
"lint:markdown": "npx -y markdownlint-cli@0.48.0 -c .github/.markdownlint.yml -i '.git' -i '__tests__' -i '.github' -i '.changeset' -i 'CODE_OF_CONDUCT.md' -i 'CHANGELOG.md' -i 'docs/**' -i 'node_modules' -i 'dist' '**/**.md' --fix",
"lint:fix": "eslint . --fix",
"test": "c8 node --import tsx --test __tests__/**/*.test.ts",
"test:watch": "c8 node --import tsx --test --watch __tests__/**/*.test.ts",
"coverage:view": "open coverage/lcov-report/index.html",
"prepare": "husky",
"version": "changeset version",
"release": "changeset publish"
},
"dependencies": {

@@ -61,2 +47,3 @@ "gpt-tokenizer": "^3.4.0"

"publishConfig": {
"provenance": true,
"access": "public"

@@ -119,3 +106,15 @@ },

]
},
"scripts": {
"start": "tsx src/bin/cli.ts",
"build": "tsc && tsdown",
"lint": "eslint . && pnpm run lint:markdown",
"lint:markdown": "npx -y markdownlint-cli@0.48.0 -c .github/.markdownlint.yml -i '.git' -i '__tests__' -i '.github' -i '.changeset' -i 'CODE_OF_CONDUCT.md' -i 'CHANGELOG.md' -i 'docs/**' -i 'node_modules' -i 'dist' '**/**.md' --fix",
"lint:fix": "eslint . --fix",
"test": "c8 node --import tsx --test __tests__/**/*.test.ts",
"test:watch": "c8 node --import tsx --test --watch __tests__/**/*.test.ts",
"coverage:view": "open coverage/lcov-report/index.html",
"version": "changeset version",
"release": "changeset publish"
}
}
+103
-6
README.md
<!-- markdownlint-disable -->
<p align="center"><h1 align="center">
<p align="center">
<h1 align="center">
tokenu
</h1>
</h1>
</p>
<p align="center">
<img width="1216" height="884" alt="tokenu-screenshot" src="https://github.com/user-attachments/assets/7186f3ec-505f-4db2-8927-81686462903b" />
</p>
<p align="center">
A Unix <code>du</code>-like CLI that counts token usage per file and directory.

@@ -15,8 +21,22 @@ </p>

<a href="https://www.npmjs.com/package/tokenu"><img src="https://badgen.net/npm/dt/tokenu" alt="downloads"/></a>
<a href="https://github.com/lirantal/tokenu/actions?workflow=CI"><img src="https://github.com/lirantal/tokenu/workflows/CI/badge.svg" alt="build"/></a>
<a href="https://github.com/lirantal/tokenu/actions/workflows/ci.yml"><img src="https://github.com/lirantal/tokenu/actions/workflows/ci.yml/badge.svg?branch=main" alt="build"/></a>
<a href="https://app.codecov.io/gh/lirantal/tokenu"><img src="https://badgen.net/codecov/c/github/lirantal/tokenu" alt="codecov"/></a>
<a href="https://snyk.io/test/github/lirantal/tokenu"><img src="https://snyk.io/test/github/lirantal/tokenu/badge.svg" alt="Known Vulnerabilities"/></a>
<a href="./SECURITY.md"><img src="https://img.shields.io/badge/Security-Responsible%20Disclosure-yellow.svg" alt="Responsible Disclosure Policy" /></a>
</p>
## Why tokenu?
When we needed to find out which folder was eating up disk space, we reached for the classic Linux `du` command. Today, when writing code with LLMs and AI agents, the question has changed: **which folder is eating up my context window?**
Every time you feed files and directories into a prompt or a coding agent, it's easy to lose control — exceed the model's limits (GPT-5.5, Claude Opus 4.6, Sonnet 4.5, etc), or just burn money on unnecessary context. **tokenu** answers this question: run a single command and get a directory tree that reveals exactly how many tokens each part of your project consumes.
### What can you do with it?
- **Understand your project's token footprint** — before pasting a folder into an LLM, see how much of the context window it will consume.
- **Find expensive files** — discover that one auto-generated JSON, lockfile, or bundle that silently blows up your token budget.
- **Budget context for AI agents** — pipe `--json` output into an autonomous agent pipeline so the agent can plan which files to read within its "memory budget".
- **Guard your context with hooks** — build a pre-read hook for tools like Claude Code that blocks or warns before loading files above a certain token threshold, preventing a single huge file from clogging the entire context.
- **Compare encodings** — quickly see how token counts differ across models (`gpt-5.4` vs `claude-4.6-opus-high`) to estimate costs before switching.
- **Audit before you ship** — verify that your published package or documentation stays within a reasonable token size for consumers who use AI-assisted workflows.
## Quick Start

@@ -30,8 +50,34 @@

Or install it globally:
### Real-World Recipes
**Spot the context hog in your project:**
```sh
npm install -g tokenu
$ tokenu -d 1 -hs .
1.2K ./src
124 ./docs
48.7K ./dist
890 ./.github
50.9K .
```
In this example `dist/` alone takes ~48K tokens — nearly the entire budget of some models. Now you know to exclude it or `.gitignore` it from your AI workflow.
**Feed an AI agent its token budget:**
```sh
$ tokenu --json -s . | your-agent --context-budget 128000
```
The agent receives structured JSON with per-directory token counts and can decide which parts of the codebase to load, staying within its context window.
**Quick-check a single file before pasting it into a prompt:**
```sh
$ tokenu -h data/large-fixture.json
23.6K data/large-fixture.json
```
If a single file costs 23K tokens you probably want to summarize it first rather than dump it raw into your prompt.
## Usage

@@ -106,2 +152,53 @@

## Install
Local install:
```sh
npm install tokenu
```
or globally install the tokenu package in your development environment:
```sh
npm install -g tokenu
```
## FAQ
### Does tokenu replace token-saving tools like rtk?
No. [rtk](https://github.com/rtk-ai/rtk) is a CLI proxy that sits between your AI agent and the shell, actively filtering and compressing command outputs to reduce token consumption by 60-90%. It rewrites commands like `git status` or `cargo test` to return compact, token-efficient output.
**tokenu** does something different: it's a read-only measurement tool that tells you how many tokens your files and directories cost *before* they enter the context window. It doesn't modify or compress anything.
Think of it this way: rtk shrinks the output your agent *receives* from shell commands, while tokenu helps you understand the token cost of the files and directories you *feed* into a prompt or agent. They complement each other — use tokenu to plan your context budget, and rtk to keep command outputs lean.
### Does tokenu reduce my token usage?
Not directly. tokenu is a diagnostic tool — it reports token counts so *you* can make informed decisions. For example, you might discover a 40K-token auto-generated file sitting in your project root and decide to exclude it from your AI workflow, saving real money and context space.
### Can AI agents use tokenu?
Yes. Run `tokenu --json` to get structured output that an autonomous agent can consume programmatically. The agent can call tokenu, inspect the per-file and per-directory token counts, and decide which parts of the codebase fit within its context window — essentially giving it a "memory budget" to plan around.
### Does tokenu use heuristics or actual tokenization?
Actual tokenization. No estimation or approximation. tokenu reads every file, feeds its content through a real tokenizer (the [`gpt-tokenizer`](https://github.com/nicolo-ribaudo/gpt-tokenizer) library), and sums the results recursively across directories. The counts you see are the same counts the model would consume.
You can also choose which tokenizer encoding to use. Different model families use different encodings, and token counts can vary between them. For example:
```sh
tokenu --model gpt-4o src/ # uses o200k_base (GPT-4o's encoding)
tokenu --encoding cl100k_base src/ # uses cl100k_base (GPT-3.5/GPT-4)
```
By default tokenu uses `o200k_base`. See the [Options](#options) table for the full list of supported encodings.
### What's a practical example of using tokenu with coding agents?
Imagine you're running Claude Code (or a similar coding agent) in a project that contains a huge `data.json` file. Without realizing it, the agent loads that file into context and it consumes your entire context window, leaving no room for actual code.
With tokenu you can build a pre-read hook: before the agent reads any file, run `tokenu` on it. If the file exceeds a token threshold (say, 10K tokens), the hook can ask for confirmation or skip the file entirely. This keeps the agent focused on what matters.
## Contributing

@@ -108,0 +205,0 @@