@huggingface/tokenizers
Advanced tools
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
A lightweight tokenizer for the Web
Run today's most used tokenizers directly in your browser or Node.js application. No heavy dependencies, no server required. Just fast, client-side tokenization compatible with thousands of models on the Hugging Face Hub. These tokenizers are also used in 🤗 Transformers.js
npm install @huggingface/tokenizers
Alternatively, you can use it via a CDN as follows:
<script type="module">
import { Tokenizer } from "https://cdn.jsdelivr.net/npm/@huggingface/tokenizers";
</script>
import { Tokenizer } from "@huggingface/tokenizers";
// Load files from the Hugging Face Hub
const modelId = "HuggingFaceTB/SmolLM3-3B";
const tokenizerJson = await fetch(`https://huggingface.co/${modelId}/resolve/main/tokenizer.json`).then((res) => res.json());
const tokenizerConfig = await fetch(`https://huggingface.co/${modelId}/resolve/main/tokenizer_config.json`).then((res) => res.json());
// Create tokenizer
const tokenizer = new Tokenizer(tokenizerJson, tokenizerConfig);
// Tokenize text
const tokens = tokenizer.tokenize("Hello World"); // ['Hello', 'ĠWorld']
const encoded = tokenizer.encode("Hello World"); // { ids: [9906, 4435], tokens: ['Hello', 'ĠWorld'], attention_mask: [1, 1] }
const decoded = tokenizer.decode(encoded.ids); // 'Hello World'
This library expects two files from Hugging Face models:
tokenizer.json - Contains the tokenizer configurationtokenizer_config.json - Contains additional metadataTokenizers.js supports Hugging Face tokenizer components:
Natural is a general natural language processing library for Node.js. It provides tokenization, stemming, classification, and other NLP tools. While it offers a broader range of NLP functionalities, it may not be as optimized for speed and efficiency in tokenization as @huggingface/tokenizers.
Compromise is a lightweight NLP library for Node.js that provides fast and easy-to-use text processing tools. It includes tokenization, part-of-speech tagging, and named entity recognition. While it is user-friendly and fast, it may not offer the same level of customization and pre-trained model support as @huggingface/tokenizers.
Wink Tokenizer is a fast and lightweight tokenizer for JavaScript. It is designed for speed and simplicity, providing basic tokenization capabilities. However, it lacks the advanced features and pre-trained model support found in @huggingface/tokenizers.
FAQs
🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers
The npm package @huggingface/tokenizers receives a total of 1,023,387 weekly downloads. As such, @huggingface/tokenizers popularity was classified as popular.
We found that @huggingface/tokenizers demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 5 open source maintainers collaborating on the project.
Security News
GPT-6 Astra hits 100% on ExploitBench and finds zero-days autonomously, while independent tests reveal scope violations and monitoring gaps.
Product
Socket can now send alerts and supply chain attack notifications to Microsoft Teams, with filters that route the right updates to each channel.
Security News
pnpm 12 rewrites the package manager in Rust, cutting install times by up to 90% while preserving pnpm 11 workflows and lockfiles.