⏳ tiktoken
tiktoken is a BPE tokeniser for use with
OpenAI's models, forked from the original tiktoken library to provide NPM bindings for Node and other JS runtimes.
The open source version of tiktoken
can be installed from NPM:
npm install @dqbd/tiktoken
Usage
Basic usage follows:
import assert from "node:assert";
import { get_encoding, encoding_for_model } from "@dqbd/tiktoken";
const enc = get_encoding("gpt2");
assert(
new TextDecoder().decode(enc.decode(enc.encode("hello world"))) ===
"hello world"
);
const enc = encoding_for_model("text-davinci-003");
const enc = encoding_for_model("gpt2", {
"<|im_start|>": 100264,
"<|im_end|>": 100265,
});
enc.free();
If desired, you can create a Tiktoken instance directly with custom ranks, special tokens and regex pattern:
import { Tiktoken } from "../pkg";
import { readFileSync } from "fs";
const encoder = new Tiktoken(
readFileSync("./ranks/gpt2.tiktoken").toString("utf-8"),
{ "<|endoftext|>": 50256, "<|im_start|>": 100264, "<|im_end|>": 100265 },
"'s|'t|'re|'ve|'m|'ll|'d| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)|\\s+"
);
Compatibility
As this is a WASM library, there might be some issues with specific runtimes. If you encounter any issues, please open an issue.
Runtime | Status | Notes |
---|
Node.js | ✅ | |
Bun | ✅ | |
Vite | ✅ | See here for notes |
Next.js | ✅ | See here for notes |
Vercel Edge Runtime | ✅ | See here for notes |
Cloudflare Workers | 🚧 | See here for caveats |
Deno | ❌ | Currently unsupported |
If you are using Vite, you will need to add both the vite-plugin-wasm
and vite-plugin-top-level-await
. Add the following to your vite.config.js
:
import wasm from "vite-plugin-wasm";
import topLevelAwait from "vite-plugin-top-level-await";
import { defineConfig } from "vite";
export default defineConfig({
plugins: [wasm(), topLevelAwait()],
});
Both API routes and /pages
are supported with the following next.config.js
configuration.
const config = {
webpack(config, { isServer, dev }) {
config.experiments = {
asyncWebAssembly: true,
topLevelAwait: true,
layers: true,
};
return config;
},
};
Usage in pages:
import { get_encoding } from "@dqbd/tiktoken";
import { useState } from "react";
const encoding = get_encoding("cl100k_base");
export default function Home() {
const [input, setInput] = useState("hello world");
const tokens = encoding.encode(input);
return (
<div>
<input
type="text"
value={input}
onChange={(e) => setInput(e.target.value)}
/>
<div>{tokens.toString()}</div>
</div>
);
}
Usage in API routes:
import { get_encoding } from "@dqbd/tiktoken";
import { NextApiRequest, NextApiResponse } from "next";
export default function handler(req: NextApiRequest, res: NextApiResponse) {
const encoding = get_encoding("cl100k_base");
const tokens = encoding.encode("hello world");
encoding.free();
return res.status(200).json({ tokens });
}
Next.js 12
Using the older Next.js 12 version? Use the @dqbd/tiktoken/bundler
or @dqbd/tiktoken/lite/bundler
instead alongside the following next.config.json
and tsconfig.json
configurations:
const config = {
webpack(config, { isServer, dev }) {
config.experiments = {
asyncWebAssembly: true,
topLevelAwait: true,
layers: true,
};
return config;
},
};
{
"compilerOptions": {
"moduleResolution": "node16"
}
}
Usage in pages:
import { get_encoding } from "@dqbd/tiktoken/bundler";
import { useState } from "react";
const encoding = get_encoding("cl100k_base");
export default function Home() {
const [input, setInput] = useState("hello world");
const tokens = encoding.encode(input);
return (
<div>
<input
type="text"
value={input}
onChange={(e) => setInput(e.target.value)}
/>
<div>{tokens.toString()}</div>
</div>
);
}
Usage in API routes:
import type { NextApiRequest, NextApiResponse } from "next";
import { get_encoding } from "@dqbd/tiktoken/bundler";
export default function handler(req: NextApiRequest, res: NextApiResponse) {
const encoding = get_encoding("cl100k_base");
const tokens = encoding.encode("hello world");
encoding.free();
res.status(200).json({ tokens });
}
Vercel Edge Runtime does support WASM modules by adding a ?module
suffix. Initialize the encoder with the following snippet:
import wasm from "@dqbd/tiktoken/tiktoken_bg.wasm?module";
import { init, get_encoding } from "@dqbd/tiktoken/init";
export const config = { runtime: "edge" };
export default async function (req: Request) {
await init((imports) => WebAssembly.instantiate(wasm, imports));
const encoder = get_encoding("cl100k_base");
const tokens = encoder.encode("hello world");
encoder.free();
return new Response(`${encoder.encode("hello world")}`);
}
Currently work in progress, investigating crashes and workarounds to compress ranks.
Similar to Vercel Edge Runtime, Cloudflare Workers must import the WASM binary file manually. However, users need to point directly at the WASM binary, including node_modules
prefix in some cases.
Add the following rule to the wrangler.toml
to upload WASM during build:
[[rules]]
globs = ["**/*.wasm"]
type = "CompiledWasm"
Initialize the encoder with the following snippet:
import wasm from "./node_modules/@dqbd/tiktoken/tiktoken_bg.wasm";
import { get_encoding, init } from "@dqbd/tiktoken/init";
export default {
async fetch() {
await init((imports) => WebAssembly.instantiate(wasm, imports));
const encoder = get_encoder("cl100k_base");
const tokens = encoder.encode("hello world");
encoder.free();
return new Response(`${tokens}`);
},
};
Acknowledgements