https://github.com/zurawiki/tiktoken-rs
Rust185
3 months ago
Ready-made tokenizer library for working with GPT and tiktoken
MIT License