shimmytok

A pure Rust tokenizer library that reads tokenizers directly from GGUF model files, requiring no Python, C++ or separate tokenizer files. It supports SPM, BPE, WPM, RWKV and UGM tokenizers tested token-for-token against llama.cpp, and offers batch encoding with optional Rayon parallelism. MIT licensed.

GitHub repo Directory import
0 shouts Added Sep 23, 2026 github.com
github.com/Michael-A-Kuykendall/shimmytok
shimmytok website preview

Features

  • pure Rust tokenizer library that reads tokenizers directly from GGUF model files Source: github.com

Links

Tags