AI Briefing#attention
The latest AI and developer news about #attention, with the original source and a short summary.
The latest AI and developer news about #attention, with the original source and a short summary.
New kernel fusion techniques for LayerNorm and RMSNorm hide up to 90% of normalization latency in GEMM kernels and achieve up to 35% speedup in attention kernels on NVIDIA B200 GPUs.
Qwen3.8-Flash-Next demonstrates superior performance compared to existing models at 1/9 the cost.
Removing the FFN nearly eliminated the performance gap with standard Transformers when the budget was reallocated.
MSLK has organized Attention, GEMM, and MoE kernel usage in one place.
Fixed-size online memory improves LLM long-term memory performance.
Selective hierarchical attention was about 17x faster than standard attention at 512K context.
Subquadratic touted a 12 million token context window with SSA and an 83 score on MRCR v2, surpassing GPT-5.5.
It was shown that transformers express formal languages more succinctly, and that verification is EXPSPACE-complete.
Sessa is a decoder architecture for long context that puts attention into a recurrent feedback path.
Based on Nova 2.0 Lite, attaching LoRA to only **o_proj** gave the best balance of efficiency and accuracy.