tiktokenman

tiktokenman

File-based token counter CLI powered by tiktoken.

Count tokens in any file before sending to an LLM API — know the cost before you pay.

Install

npm i -g tiktokenman

Or run directly:

npx tiktokenman count ./file.txt

Usage

Single file

$ tiktokenman count ./README.md
README.md    342 tokens (gpt-4o)

Multiple files / glob patterns

$ tiktokenman count "src/**/*.ts"
 src/cli.ts              83 tokens
 src/commands/count.ts  559 tokens
 src/index.ts           176 tokens
────────────────────────────────────
 Total                  818 tokens (gpt-4o)

Specify model

$ tiktokenman count ./file.txt --model gpt-4o-mini

Supported models: gpt-4o (default), gpt-4o-mini, gpt-4, gpt-3.5-turbo

JSON output

$ tiktokenman count "src/**/*.ts" --json

{
  "model": "gpt-4o",
  "files": [
    { "file": "src/cli.ts", "tokens": 83 },
    { "file": "src/commands/count.ts", "tokens": 559 },
    { "file": "src/index.ts", "tokens": 176 }
  ],
  "total": 818
}

Options

Option	Alias	Description	Default
--model	-m	Tokenizer model	gpt-4o
--json		Output as JSON	false
--help	-h	Show help
--version	-v	Show version

Programmatic API

import { countTokens, countFileTokens } from "tiktokenman";

// Count tokens from text
const tokens = countTokens("Hello, world!", "gpt-4o");
console.log(tokens); // 4

// Count tokens from a file
const result = countFileTokens("./README.md");
console.log(result); // { file: "./README.md", tokens: 342, model: "gpt-4o" }

License

MIT