@tokenometer/core

@tokenometer/core

Core library powering tokenometer: tokenizer dispatch, format converters, versioned cost rate matrix, and an empirical-mode countTokens adapter for Anthropic, OpenAI, and Google.

Live playground · Source · MIT

If you just want a CLI, npm install -g tokenometer. This package is for programmatic use.

API

import {
  tokenize,
  tokenizeMatrix,
  tokenizeEmpirical,
  tokenizeMatrixEmpirical,
  countTokens,
  toFormat,
  isFormat,
  allFormats,
  KNOWN_MODELS,
  RATES,
  RATES_VERSION,
  getModel,
  getRate,
} from '@tokenometer/core';

Offline (deterministic, no API key)

const result = tokenize({
  prompt: '{"hello": "world"}',
  format: 'yaml',
  modelId: 'claude-opus-4-7',
});
// {
//   model: 'claude-opus-4-7',
//   provider: 'anthropic',
//   format: 'yaml',
//   tokenizer: 'cl100k_base',
//   inputTokens: 12,
//   inputCost: 0.00018,
//   approximate: true   // ← Anthropic does not publish a public Claude 3+ tokenizer
// }

Empirical (real provider counts, free)

const result = await tokenizeEmpirical({
  prompt: '{"hello": "world"}',
  format: 'yaml',
  modelId: 'claude-opus-4-7',
  env: { anthropicApiKey: process.env.ANTHROPIC_API_KEY! },
});
// approximate: false  ← uses Anthropic's messages.countTokens

Rate table

RATES is a Record<modelId, { inputPer1k, outputPer1k, cachedInputPer1k? }>. RATES_VERSION ships as a date string so consumers can pin or audit.

License

MIT