Marcel Røed Releases Gigatoken on GitHub — Rust Tokenizer Runs 280–1,353x Faster Than HuggingFace, up to 681x Over Tiktoken
Summary
Marcel Røed published Gigatoken, an open-source Rust tokenizer positioned as a drop-in replacement for HuggingFace tokenizers, claiming 280x–1,353x speedups over HF and 50x–681x over Tiktoken via SIMD, aggressive pretoken caching and minimizing Python round-trips. On GPT-2 with an AMD EPYC CPU, Gigatoken hit 24.53 GB/s versus HuggingFace's 24.8 MB/s. Trending on r/LocalLLaMA today.
Originally reported by github.com
Read the original article →Original headline: Marcel Røed Releases Gigatoken on GitHub — Rust Tokenizer Runs 280–1,353x Faster Than HuggingFace, up to 681x Over Tiktoken