@lenml/tokenizer-claude1
Advanced tools
a tokenizer.
based on
@lenml/tokenizers
import { fromPreTrained } from "@lenml/tokenizer-claude1";
const tokenizer = fromPreTrained();
console.log(
"encode()",
tokenizer.encode("Hello, my dog is cute", null, {
add_special_tokens: true,
})
);
console.log(
"_encode_text",
tokenizer._encode_text("Hello, my dog is cute")
);
Complete api parameters and usage can be found in transformer.js tokenizers document
Apache-2.0
FAQs
claude1 tokenizer for NodeJS/Browser
We found that @lenml/tokenizer-claude1 demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Security News
Two compromised GitHub Actions were re-enabled with malicious tags intact, exposing thousands of downstream repositories to Mini Shai-Hulud.
Research
/Security News
A malicious Firefox extension fetches its payload after installation to evade detection, steal Google session cookies, and automate account takeover.
Research
/Security News
The compromise affects MemTensor's MemOS, an open source memory framework for large language models (LLMs) and AI agents. Both npm package @memtensor/memos-cloud-openclaw-plugin and the PyPI package MemoryOS are compromised. They drop cross-platform Go binaries that exfiltrate developer secrets.