AI BriefingWall Attention (GitHub repository)
Key point
Wall Attention has been released, applying per-channel multiplicative decay to support efficient training and inference.
Details
Wall Attention is a new attention variant that incorporates per-channel, per-timestep multiplicative decay into the QK dot product. Unlike conventional attention, which treats all channels equally when computing the score between two positions, Wall Attention applies an independent, content-dependent forgetting rate to each channel.
This approach generalizes scalar gating (FoX) or RoPE-style decay to the full channel dimension. Setting the decay coefficient $g$ to 0 makes it behave identically to conventional Vanilla Softmax Attention.
The repository provides two core kernels for practical use.
- Training / Prefill (
wall_attn): A fused Triton kernel that supports analytic gradients for $q, k, v, g$. - Decode (
wall_attn_decode): Reads a pre-rescaled KV cache, enabling efficient inference during token generation without needing to recompute the entire prefix.
Key features include support for GQA (Grouped Query Attention), Attention Sink, Sliding Window, and Varlen Packing. In particular, the pre-rescaled decode cache is numerically stable even at long context lengths, supporting cheap autoregressive generation.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Hierarchical Memory Architecture Needed to Resolve AI Inference Bottlenecks