Diff-Transformer V2 Released

Differential Transformer V2 is a new architecture that improves upon the structural limitations of the existing V1, boosting both inference efficiency and computational performance at the same time.

Key Improvements:

• Decoding speed optimization: The number of Query heads is doubled, while the number of Key-Value heads (KV heads) is kept unchanged. This design increases arithmetic intensity without increasing the memory bandwidth load, which is the bottleneck in LLM decoding.
• No custom kernel required: Because the head dimensions are designed to be aligned, standard FlashAttention can be used as is, without a separate custom attention kernel, making implementation easier.
• Softmax Magnitude Constraint resolved: The imbalance issue in context RMS (Root Mean Square) that occurred in V1 has been resolved through mathematical design, allowing the model to learn a more stable attention distribution.

Performance and Applications:

• The reduction in throughput during the pretraining stage is minimal.
• When combined with techniques such as YOCO for long-sequence prefilling, efficiency can be maximized with linear time complexity.