Lattice of Latents architecture lowers Transformer perplexity to 14.40
·2026.11.18 07:17
Key point
The Lattice of Latents adds horizontal recurrent streams to Transformers, reducing validation perplexity from 15.41 to 14.40 at 406M parameters.
Details
- Introduces the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently alongside the vertical residual stream to increase computational capacity.
- On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 compared to a parameter-matched Transformer.
- The architecture composes with prior residual-stream enhancements like hyper-connections and input residuals, further lowering perplexity to 13.61.
- Gains extend to recurrent backbones such as Gated DeltaNet and Mamba-2, where Lattice variants outperform pure baselines with 4x larger native states.
- At 10B tokens, the 406M LLLM achieves lower perplexity than a 609M Transformer while using 45% smaller KV cache due to fewer layers.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.