Delta Attention Residuals Released

Delta Attention Residuals, which upgrades existing residual connections to maximize a model's routing efficiency, has been released. This technique learns which of the previous layers to route, solving the chronic problem of existing methods known as 'routing collapse.'

Key features and results are as follows:

• Delta-based Routing: Instead of the accumulated hidden state, it routes the delta value ($v_i = h_{i+1} - h_i$), which is the contribution of each sublayer. This raised the maximum attention weight from about 0.2 to 0.6, achieving sharper cross-layer routing.
• Performance and Stability: On the 7.6B model, it reduced validation PPL by 8.2%, demonstrating consistent performance improvements across scales from 220M to 7.6B. In particular, it overcomes the problem where existing Attention Residuals methods degrade in performance at large model scales.
• Application to Pretrained Models: By adopting a Zero-init routing method, existing models such as Qwen3-0.6B can be easily upgraded through fine-tuning alone.
• Very Low Overhead: The parameter increase is under 0.01%, and memory usage increases by only about 3%, making it very lightweight and fast.