tiktoken-node
Advanced tools
Readme
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
const tiktoken = require('tiktoken-node')
let enc = tiktoken.getEncoding("gpt2")
console.assert(enc.decode(enc.encode("hello world")) == "hello world")
// To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encodingForModel("text-davinci-003")
The open source version of tiktoken-node can be installed from npm:
npm install tiktoken-node
FAQs
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
The npm package tiktoken-node receives a total of 10,575 weekly downloads. As such, tiktoken-node popularity was classified as popular.
We found that tiktoken-node demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
A hospital in Mobile, Alabama, agreed to a settlement in a landmark ransomware death lawsuit, but is now reportedly reconsidering the agreement and refusing to pay.
Security News
A new report explores how advancements in LLMs are enhancing cyber threats, including polymorphic malware, personalized spearphishing, and the risk of hijacking customer service bots.
Security News
ESLint has approved an RFC that adds support for TypeScript configuration files, which is aimed at improving the developer experience and recognizing changes in the evolving JavaScript ecosystem.