AI BriefingAttention Mass Predicts Context Rot in Retrieval: Competing Statements Hurt More Than Length
·2026.11.05 06:08
Key point
Competing statements in context cause 9-110x higher loss than neutral filler, while length alone explains only 2-8% of variance.
Details
- Attention mass on evidence (a*) explains 35-58% of loss variance across 126,473 trials, whereas log length explains only 2-8%.
- Neutral filler is nearly free, maintaining 0.97-1.00 accuracy up to 32K tokens on Qwen2.5-1.5B, while competing statements drop accuracy to 0.11-0.21.
- A single competing statement multiplies loss by 9 to 110x, indicating that semantic competition, not context length, drives retrieval failure.
- Masking filler (as in Du et al., 2025) costs 46 accuracy points at 32K in retrieval, while visible neutral filler costs none, suggesting length claims require visible controls.
- Study covers five open models (0.5B to 3B) across four families; attention result is predictive, not causal, and limited to one synthetic task.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Study finds LLMs break 32-51 working PHP files during modernization
Training 1,000 Weights in Gemma 3 270M Unlocks Hidden Sorting and JSON SkillsQualcomm Launches Cloud AI 100 Ultra Inference Card with 128GB Memory and Up to 870 TOPS