AI BriefingMuir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout (Google Research, 10M tokens in 25MB state) [R]
·2026.11.07 04:10
Key point
The model resolves slow-channel glare and write sensitivity bottlenecks to train 2.7x faster than Splash Attention at 32K context.
Details
- Google Research releases Muir, a linear attention model that compresses 10M tokens of history into a fixed 25 MB state, avoiding the 2.1 TB KV cache required by standard 1.3B Transformers.
- Muir introduces matched-filter readout to equalize slow-decaying rows, relaxing write erasure by 17% and extending the lookback horizon by 60% compared to Gated DeltaNet-2 (GDN-2).
- A fused rank-512 projection cuts gate parameters by 4.5x, allowing the FFN width to increase by 30.4% while maintaining a strict parameter budget.
- The model achieves 90.0% zero-shot passkey accuracy at 1M tokens and streams 10M tokens without perplexity drift, whereas GDN-2 collapses past 4K-8K tokens and Transformers run out of memory past 64K.
- Performance benchmarks show Muir beats GDN-2 and Transformers on perplexity and zero-shot reasoning, with 1.2x token efficiency versus GDN-2 and 2.7x faster training speed than Splash Attention at 32K context.
- Recall metrics improve significantly over GDN-2, including RULER 5-hop Variable Tracking root retention rising from 28.4% to 48.0% and early-depth recall at 8K increasing from 1% to 31%.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Memorilla Compresses LLM Memory into 16 Tokens for Better Long-Context Performance