AI BriefingParallax: A New Attention Technique
Key point
Parallax attention has been proposed, overcoming the limitations of existing LLA while achieving performance on par with FlashAttention.
Details
Parallax is a parameterized local linear attention (LLA) mechanism that improves the attention structure, the core operation of LLMs.
Existing LLA was difficult to apply to LLM pretraining due to numerical stability and scalability issues, but Parallax solves this by removing the numerical solver and instead training an additional query-like projector that explores KV covariance.
Key features are as follows:
- Hardware-aware algorithm: It achieves higher arithmetic intensity than FlashAttention, shifting attention into the compute-bound regime.
- Performance: The prototype decode kernel matches or outperforms FlashAttention 2/3 across various batch sizes and context lengths.
- Scalability: Pretraining at the 0.6B and 1.7B scales demonstrated improvements in perplexity and downstream benchmark performance.
Additionally, the discovery that combining it with the Muon optimizer maximizes Parallax's capabilities demonstrates a strong co-design effect between architecture and optimizer.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Proposal for 'Infinite-Parameter LLM' Architecture That Generates and Adapts Weights with Real-Time DataPPinterest Unveils 3-Tower Engagement Co-Train Model: Achieves 1% CTR Lift and 30ms Latency