Gigatoken released, processing gigabytes per second

Developed to solve the text tokenization bottleneck that occurs during large language model (LLM) training, gigatoken is a Rust-based tokenizer aiming for gigabyte-per-second (GB/s) throughput.

Key features are as follows:

• Drop-in Replacement: It can replace existing HuggingFace tokenizers or tiktoken code with minimal modifications.
• Overwhelming Throughput: Benchmark results showed performance about 989x faster than HuggingFace on the GPT-2 tokenizer, and about 457x faster on Llama 3.
• Optimized API: To reduce Python overhead, it provides a dedicated API where the Rust implementation reads data directly to maximize parallelism.
• Compatibility and License: It supports a variety of CPU hardware and is released under the MIT License, allowing free commercial use.

Unlike existing tools that remain at MB/s-level performance, gigatoken can dramatically improve the efficiency of the entire pipeline when processing large-scale corpora.