Researchers introduce 'byteification' to retrofit subword LLMs into byte-level models with minimal training cost

Byteification Methodology

Researchers propose byteification, a technique to retrofit existing subword-based LLMs into byte-level models without random initialization. This approach addresses limitations of subword tokenization, such as fine-grained information loss, prompt bias, and English-centricity, while avoiding the high cost of training new byte-level models from scratch. The process uses less than 1% of a typical pretraining budget, totaling 49.1B tokens.

Architecture and Training

The method employs a Latent Tokenizer Language Model (LTLM) style architecture, aggregating byte streams into patches for Transformer processing and then depooling them back to bytes. Key architectural innovations include:

• Non-causal boundary prediction: Unlike previous LTLMs, the boundary predictor uses future context to mimic subword tokenizer behavior, resolving the mismatch between subword and latent tokenization expressivity.
• Two-stage training:
  • Stage 1: The global model is frozen while local encoders, decoders, and boundary predictors are trained to emulate the source subword model's behavior.
  • Stage 2: The entire model is trained end-to-end to leverage byte-level information and increase compression ratios.

Performance and Results

The resulting models—Bolmo 7B (based on Olmo 3 7B), Bolmo 1B (OLMo 2 1B), and Bwen 8B (Qwen3 8B Base)—demonstrate significant improvements over existing byte-level LLMs:

• Bolmo 7B shows a +16.5% absolute performance gain in STEM tasks compared to the randomly initialized BLT 7B.
• Bwen 8B outperforms Bolmo 7B and reaches performance close to and sometimes surpassing the source Qwen model.
• Byteified models substantially surpass their subword counterparts in character understanding tasks, a capability previously difficult to achieve with byte-level models.

Ecosystem Compatibility

A major advantage is the ability to reuse existing subword LLM ecosystem components. By applying adaptation techniques, post-trained checkpoints from source models can be merged into byteified models to inject capabilities without additional training, significantly reducing iteration time.