@dqbd/tiktoken - npm Package Compare versions

Comparing version 1.0.0-alpha.1 to 1.0.0-alpha.2

package.json

 {
   "name": "@dqbd/tiktoken",
-  "version": "1.0.0-alpha.1",
+  "version": "1.0.0-alpha.2",
   "description": "Javascript bindings for tiktoken",
   "license": "MIT",
-  "scripts": {
-    "build": "run-s build:*",
-    "build:cleanup": "rm -rf dist/",
-    "build:rank": "tsx scripts/inline_ranks.ts",
-    "build:wasm": "run-s wasm:*",
-    "build:postprocess": "tsx scripts/post_process.ts",
-    "wasm:bundler": "wasm-pack build --target bundler --release --out-dir dist && rm -rf dist/.gitignore dist/README.md dist/package.json",
-    "wasm:node": "wasm-pack build --target nodejs --release --out-dir dist/node && rm -rf dist/node/.gitignore dist/node/README.md dist/node/package.json",
-    "test": "yarn vitest"
-  },
   "repository": {
@@ -21,10 +11,55 @@ "type": "git",
   "dependencies": {},
-  "devDependencies": {
-    "@types/node": "^18.14.4",
-    "npm-run-all": "^4.1.5",
-    "ts-morph": "^17.0.1",
-    "tsx": "^3.12.3",
-    "typescript": "^4.9.5",
-    "vitest": "^0.28.5"
+  "files": [
+    "**/*"
+  ],
+  "main": "tiktoken.cjs",
+  "types": "tiktoken.d.ts",
+  "exports": {
+    ".": {
+      "types": "./tiktoken.d.ts",
+      "node": "./tiktoken.cjs",
+      "default": "./tiktoken.js"
+    },
+    "./bundler": {
+      "types": "./bundler.d.ts",
+      "default": "./bundler.js"
+    },
+    "./init": {
+      "types": "./init.d.ts",
+      "node": "./init.cjs",
+      "default": "./init.js"
+    },
+    "./load": {
+      "types": "./load.d.ts",
+      "node": "./load.cjs",
+      "default": "./load.js"
+    },
+    "./tiktoken_bg.wasm": {
+      "types": "./tiktoken_bg.wasm.d.ts",
+      "default": "./tiktoken_bg.wasm"
+    },
+    "./lite": {
+      "types": "./lite/tiktoken.d.ts",
+      "node": "./lite/tiktoken.cjs",
+      "default": "./lite/tiktoken.js"
+    },
+    "./lite/bundler": {
+      "types": "./lite/bundler.d.ts",
+      "default": "./lite/bundler.js"
+    },
+    "./lite/init": {
+      "types": "./lite/init.d.ts",
+      "node": "./lite/init.cjs",
+      "default": "./lite/init.js"
+    },
+    "./lite/load": {
+      "types": "./lite/load.d.ts",
+      "node": "./lite/load.cjs",
+      "default": "./lite/load.js"
+    },
+    "./lite/tiktoken_bg.wasm": {
+      "types": "./lite/tiktoken_bg.wasm.d.ts",
+      "default": "./lite/tiktoken_bg.wasm"
+    }
   }
-}
+}

README.md

@@ -56,11 +56,11 @@ # ⏳ tiktoken
 
-| Runtime             | Status | Notes                                      |
-| ------------------- | ------ | ------------------------------------------ |
-| Node.js             | ✅     |                                            |
-| Bun                 | ✅     |                                            |
-| Vite                | ✅     | See [here](#vite) for notes                |
-| Next.js             | ✅     | See [here](#nextjs) for notes              |
-| Vercel Edge Runtime | ✅     | See [here](#vercel-edge-runtime) for notes |
-| Cloudflare Workers  | 🚧     | Untested                                   |
-| Deno                | ❌     | Currently unsupported                      |
+| Runtime             | Status | Notes                                       |
+| ------------------- | ------ | ------------------------------------------- |
+| Node.js             | ✅     |                                             |
+| Bun                 | ✅     |                                             |
+| Vite                | ✅     | See [here](#vite) for notes                 |
+| Next.js             | ✅     | See [here](#nextjs) for notes               |
+| Vercel Edge Runtime | ✅     | See [here](#vercel-edge-runtime) for notes  |
+| Cloudflare Workers  | 🚧     | See [here](#cloudflare-workers) for caveats |
+| Deno                | ❌     | Currently unsupported                       |
 
@@ -133,4 +133,34 @@ ### [Vite](#vite)
 
+### [Cloudflare Workers](#cloudflare-workers)
+
+Similar to Vercel Edge Runtime, Cloudflare Workers must import the WASM binary file manually. However, users need to point directly at the WASM binary, including `node_modules` prefix in some cases.
+
+Add the following rule to the `wrangler.toml` to upload WASM during build:
+
+```toml
+[[rules]]
+globs = ["**/*.wasm"]
+type = "CompiledWasm"
+```
+
+Initialize the encoder with the following snippet:
+
+```javascript
+import wasm from "./node_modules/@dqbd/tiktoken/tiktoken_bg.wasm";
+import { get_encoding, init } from "@dqbd/tiktoken/init";
+
+export default {
+  async fetch() {
+    await init((imports) => WebAssembly.instantiate(wasm, imports));
+    const encoder = get_encoder("cl100k_base");
+    const tokens = encoder.encode("hello world");
+    encoder.free();
+    return new Response(`${tokens}`);
+  },
+};
+```
+
 ## Acknowledgements
 
 - https://github.com/zurawiki/tiktoken-rs
+

New alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Native code

Supply chain risk

Contains native code (e.g., compiled binaries or shared libraries). Including native code can obscure malicious behavior.

Found 2 instances in 1 package

Major refactor

Supply chain risk

Package has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.

Found 1 instance in 1 package

Network access

Supply chain risk

This module accesses the network.

Found 1 instance in 1 package

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

Fixed alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Deprecated

Maintenance

The maintainer of the package marked it as deprecated. This could indicate that a single version should not be used, or that the package is no longer maintained and any new vulnerabilities will not be fixed.

Found 1 instance in 1 package

Native code

Supply chain risk

Contains native code (e.g., compiled binaries or shared libraries). Including native code can obscure malicious behavior.

Found 1 instance in 1 package

Major refactor

Supply chain risk

Package has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.

Found 1 instance in 1 package

Dynamic require

Supply chain risk

Dynamic require can indicate the package is performing dangerous or unsafe dynamic code execution.

Found 1 instance in 1 package

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

Improved metrics

Total package byte prevSize

5048212

increased by19.34%

Dev dependency count

0

decreased by-100%

Number of package files

35

increased by66.67%

Lines of code

2408

increased by73.61%

Number of medium maintenance alerts

0

decreased by-100%

Number of lines in readme file

165

increased by22.22%

Worsened metrics

Number of medium supply chain risk alerts

7

increased by250%

No dependency changes