AI BriefingA New Axis for Depth Scaling
Key point
MoDA improves depth scaling, boosting both performance and efficiency in a 1.5B model.
Details
Targeting the problem where information formed in shallow layers gets diluted by residual updates as LLMs get deeper, this work proposes Mixture-of-Depths Attention (MoDA).
By having each attention head reference both the current layer's KV pairs and depth KV pairs from earlier layers, it mitigates the signal loss caused by increasing depth.
It also presents a hardware-efficient algorithm that solves the non-contiguous memory access problem, achieving 97.3% of FlashAttention-2's efficiency at a 64K sequence length.
In experiments on a 1.5B-parameter model, it showed consistent improvements over strong baselines.
- 0.2 improvement in average perplexity across 10 validation benchmarks
- 2.11% improvement in average performance across 10 downstream tasks
- Very small additional FLOPs overhead of 3.7%
- Better performance when combined with post-norm than with pre-norm
In conclusion, MoDA is presented as a promising primitive for LLM designs that push depth further.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.