Controlled Study on Attention-Only Transformers

A controlled experiment compared the Attention-Only Transformer, specifically the Simple Attention Network (SAN) which uses only attention without a Feed-Forward Network (FFN), against the standard Transformer. The researchers matched parameter count, training FLOPs, and layer depth, training on up to 105 billion tokens across models ranging from 6 million to 87 million parameters.

Simply removing the FFN from its original position resulted in the standard Transformer performing better. At the same depth, the loss difference was 0.47 nats, and at the same training FLOPs, it was 0.26 nats. However, when the saved budget was reinvested into attention layer depth, the gap at the same parameter count shrank to 0.006 nats.

SANs were stronger on tasks requiring context-based answers but weaker on tasks requiring knowledge stored in model weights. The researchers explained this difference as a parametric recall issue; the predicted gap of 0.02–0.05 nats on knowledge-dense web text was measured at 0.040 nats in FineWeb-Edu experiments.

Weight analysis revealed that Q/K routing matrices structured rapidly while the rank of content matrices increased slowly. The key factor that kept 48-layer Attention-Only models trainable was not FFN or residual gating, but QK-normalization.