Cartesia Research H-Net Achieves 1.3x Compute Efficiency Over BPE Transformers

H-Net replaces static BPE tokenization with dynamic, end-to-end hierarchical chunking, achieving 1.3x compute efficiency on validation loss and 2x efficiency on downstream evaluations.
Code performance improves significantly, with compute multipliers reaching a median of 2.04x for in-distribution languages and 3.10x for out-of-distribution languages like Prolog and Haskell.
A 2-stage H-Net model matches the performance of SmolLM3 (3B parameters) using approximately 10x less compute and data, requiring only ~1T tokens versus SmolLM3's ~11T tokens.
The architecture generalizes to non-language domains, showing promise in biology and vision, with ongoing research demonstrating emergent behaviors in DNA sequence modeling and dynamic spatial chunking in vision tasks.

Architecture and Methodology

Five-module design: encoder, learned chunking, main network, de-chunking, and decoder, allowing recursive nesting and dynamic abstraction levels.
Baseline comparisons use matched FLOPs and parameters, with H-Net's outer networks accounting for approximately 43% of total FLOPs compared to 8% for standard embedding layers.
Training utilizes a byte-level mixture (70% English, 20% code, 5% math, 5% multilingual) and Muon optimizer with WSD learning rate schedules.

In-context tokenization allows the model to adapt chunking during inference, grouping repeated motifs and new concepts dynamically.
Biological analysis reveals emergent alignment with codon borders in coding regions and distinct segmentation for exons versus introns.
Vision extension demonstrates the ability to infer 3D structure from 2D projections and consolidate low-information areas without enforcing local spatial grouping.