@brndnjs/tokencount

@brndnjs/tokencount

Count tokens from a file or stdin, wc-style. Ships a tokencount command (and a tc alias) plus a small library API.

Install

npm install -g @brndnjs/tokencount   # for the CLI
npm install @brndnjs/tokencount      # for the library

CLI

# single file: count then name
$ tc README.md
421 README.md

# stdin: just the count (pipes cleanly)
$ cat README.md | tc
421

# multiple files: per-file lines plus a total
$ tc a.md b.md
421 a.md
 88 b.md
509 total

- reads standard input. -h/--help and -v/--version behave as usual. Exit codes: 0 success, 1 a file could not be read (reported wc-style on stderr, other files still counted), 2 a bad argument.

The tc alias collides with iproute2's traffic-control command on Linux; whichever comes first on your $PATH wins. macOS has no system tc.

Library

import { countTokens, countTokensInStream } from "@brndnjs/tokencount";

countTokens("hello world"); // => 2
await countTokensInStream(process.stdin); // constant-memory streaming count

What counts as a token

Tokens are counted with the o200k_base encoding (GPT-4o / o-series), via gpt-tokenizer. There is no accurate offline Claude tokenizer, so this is an approximation for non-OpenAI models, but it is the de-facto standard for a local token count.