Transformer Performance: Mechanism Diversity Is Key Over Placement

A paper titled 'Placement Is Free, Composition Is Not', published by a research team including VIDRAFT AI Research, empirically demonstrated that Composition of mechanisms is decisive for performance over Placement in Transformer architecture design.

Key Findings

• Absence of Placement Effect: When 7 attention mechanisms were arranged in a Latin Square in a 49-layer model, the performance difference due to simple order changes was +0.16%, showing no statistical significance. This suggests that stack order tuning is meaningless.
• Importance of Mechanism Diversity: Performance dropped by -1.68% when unified into a single mechanism, and also dropped by -0.59% when concentrated on specific mechanisms. The primary cause of performance improvement was securing mechanism diversity.
• Contribution of SSM: Leave-one-out analysis showed that removing the Mamba-2 (SSM) family resulted in the largest performance drop of -2.14%. Introducing mechanisms from other families, such as SSMs, contributes more to performance than differences between attention variants.

Experiment and Model Information

• Model: Aether 6.59B MoE (2.98B active parameters, 49 layers, 25 experts top-7)
• Experiment Scale: Controlled experiments were conducted at 700M and 1.5B parameter scales. When scaled to 1.5B, the gap expanded to +2.63% when using a single mechanism, confirming that the efficiency of heterogeneous stacks increases with capacity.
• Cost Analysis: At a context length of 32K, sliding attention is approximately 1.8x faster than full attention, with higher overall system processing efficiency.

Practical Implications

1. Design Strategy: Mixing mechanisms from different families, such as SSM/linear recurrence, is more effective for performance improvement than optimizing stack order or adding attention variants.
2. Context Length Consideration: While full attention is faster for short contexts (2K), sliding attention-based designs are advantageous for long contexts (32K).
3. Reproducibility: Weights, source code, training logs, and intermediate checkpoints are released under the Apache 2.0 license to ensure transparency.