Compute-Optimal Tokenization

Researchers trained about 1,300 models to recompute compression-aware neural scaling laws. The key variable was not token count but bytes per token, and the optimal allocation of compute resources varied depending on information density.

The existing Chinchilla rule of thumb of 'about 20 tokens per parameter' turned out to be tied to a specific subword tokenizer, particularly the Byte-Pair Encoding(BPE) family. Counting only tokens misses differences in how much information a single token carries, which distorts scaling by that same amount.

Tokenization is not a fixed preprocessing step but a dynamic scaling variable. The optimal compression rate changes with the compute budget, and larger FLOP budgets required lower compression. Ultimately, a tokenizer-agnostic scaling law that aligns data scale on a bytes basis offers a reference point for designing multilingual and multimodal models.