Nature paper presents 'byteification' method to convert LLMs to byte-level processing using 49.1B tokens

Details

Researchers at Nature published a method called 'byteification' that converts subword language models to byte-level ones using 49.1 billion tokens, less than 1% of a typical pretraining budget.
Bolmo 7B, derived from Olmo 3 7B, showed a 16.5% absolute improvement in STEM tasks compared to the BLT 7B baseline while keeping general-benchmark performance close to the original subword model.
The conversion process involves two stages: a 9.8B-token first stage with the global model frozen and a 39.3B-token second stage.
The method was applied to four open models: Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base, and Llama 3 8B, resulting in Bolmo 7B, Bolmo 1B, Bwen 8B, and Blama 8B respectively.
Technical implementation uses non-causal boundary prediction with one byte of future context and doubles the byte vocabulary from 256 to 512 by fusing boundary symbols with bytes.
Parameter counts changed minimally, with Bolmo 7B increasing by 4.5% and Bolmo 1B decreasing by 0.7%.
The resulting models operate at practical inference speeds and remain compatible with existing serving stacks, addressing the issue that subword tokenization obscures fine-grained information in scientific data like code and biological sequences.

Key point