AI BriefingSATFormer: An Efficient Representation Reuse Architecture
Key point
SATFormer is proposed, improving efficiency by selectively reusing early-layer representations on a per-token and per-head basis.
Details
Recent Transformer variants have used a method of exposing early-layer representations to later layers in order to improve information flow. However, this has the drawback of increasing computation and memory costs.
Instead of uniformly copying early features into every layer, SATFormer introduces a token-wise, head-wise, context-dependent gate that learns when and where to access the value stream of the first layer.
Key results:
- Improved performance: Improved validation loss compared to Transformer and ResFormer on models ranging from 130M to 1.3B scale.
- Retrieval performance: Slightly outperforms MUDDFormer on retrieval-intensive benchmarks, scoring 1.5 points higher than ResFormer on average.
- High throughput: Maintains Transformer/ResFormer-level throughput, much faster than HyperConnections or MUDDFormer.
- Mechanism analysis: Confirmed that the gate operates not as a simple residual shortcut, but in a way that is sparse, depth-dependent, and head-specific.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Microsoft's Full-bandwidth Transformer
Neutrino-1 8B