llama-tokenizer-js
Advanced tools
Comparing version 1.2.0 to 1.2.1
{ | ||
"name": "llama-tokenizer-js", | ||
"version": "1.2.0", | ||
"version": "1.2.1", | ||
"description": "JS tokenizer for LLaMA-based LLMs", | ||
"main": "llama-tokenizer.js", | ||
"types": "types.d.ts", | ||
"scripts": { | ||
@@ -7,0 +8,0 @@ "test": "node test-llama-tokenizer.js" |
@@ -9,3 +9,3 @@ # 🦙 llama-tokenizer-js 🦙 | ||
Developed by [belladore.ai](https://belladore.ai) | ||
Developed by [belladore.ai](https://belladore.ai) with contributions from [xenova](https://github.com/xenova), [blaze2004](https://github.com/blaze2004), [imoneoi](https://github.com/imoneoi) and [ConProgramming](https://github.com/ConProgramming). | ||
@@ -114,2 +114,18 @@ ## Features | ||
const tokenizer = new LlamaTokenizer(custom_vocab, custom_merge_data); | ||
``` | ||
``` | ||
## Repo maintenance | ||
Release steps: | ||
1. node test-llama-tokenizer.js | ||
2. open test.html | ||
3. do you need to update this README? | ||
4. bump version number in root package.json | ||
5. push tokenizer changes to github | ||
6. npm publish --dry-run | ||
7. npm publish | ||
8. bump version number in example-demo/package.json | ||
9. cd example-demo && npm run build && live-server | ||
10. push example demo changes to github | ||
11. create release in github |
686673
129