js-tiktoken - npm Package Compare versions

Comparing version 1.0.18 to 1.0.19

package.json

 {
   "name": "js-tiktoken",
-  "version": "1.0.18",
+  "version": "1.0.19",
   "description": "JavaScript port of tiktoken",
@@ -5,0 +5,0 @@ "license": "MIT",

README.md

@@ -12,7 +12,34 @@ # ⏳ js-tiktoken
 
-## Usage
+## Lite
 
-Basic usage follows, which includes all the OpenAI encoders and ranks:
+You can only load the ranks you need, which will significantly reduce the bundle size:
 
 ```typescript
+import { Tiktoken } from "js-tiktoken/lite";
+import o200k_base from "js-tiktoken/ranks/o200k_base";
+
+const enc = new Tiktoken(o200k_base);
+assert(enc.decode(enc.encode("hello world")) === "hello world");
+```
+
+Alternatively, encodings can be loaded dynamically from our CDN hosted on Cloudflare Pages.
+
+```typescript
+import { Tiktoken } from "js-tiktoken/lite";
+
+const res = await fetch(`https://tiktoken.pages.dev/js/o200k_base.json`);
+const o200k_base = await res.json();
+
+const enc = new Tiktoken(o200k_base);
+assert(enc.decode(enc.encode("hello world")) === "hello world");
+```
+
+## Full usage
+
+If you need all the OpenAI tokenizers, you can import the entire library:
+
+> [!CAUTION]
+> This will include all the OpenAI tokenizers, which may significantly increase the bundle size. See
+
+```typescript
 import assert from "node:assert";
@@ -19,0 +46,0 @@ import { getEncoding, encodingForModel } from "js-tiktoken";

No alert changes

Improved metrics

Total package byte prevSize

22428740

Number of lines in readme file

50

increased by117.39%

No dependency changes