Reading Notes on the DeepSeek-V4 Paper - Roh Jeongseok

DeepSeek-V4 scaled up to 1.6T compared to V3's 600B, achieving major innovations on both the architecture and infrastructure fronts.

Core Architectural Changes

• Introduction of Sparse Attention: Combined sliding window, compressed full attention, and Compressed Sparse Attention using a Lightning Indexer to maximize efficiency.
• mHC (Manifold-Constrained Hyper-Connections): Applied hyper-connections stabilized on a manifold to relax the constraints of residual connections.
• Muon Optimizer: Adopted instead of Adam to improve training speed and data efficiency.
• Removal of MLA: Dropped the previously iconic MLA in favor of simple Multi-Query Attention.

Key Advantages and Results

• Reduced Long-Context Costs: Although the model scale grew 2.5–3x, per-token compute was cut to about 27% and KV cache memory to about 10%.
• Integrated Design from the Pretraining Stage: Long-context training, Sparse Attention, and MXFP4 (4-bit) quantization were baked in from the pretraining stage rather than added afterward.
• Infrastructure Optimization: Pushed hardware efficiency to the limit through MoE communication/computation overlap, mega-kernels, and TileLang contributions.

Limitations and Challenges

• Training Instability: The training process went through considerable trial and error, requiring complex mechanisms such as MoE gating fixes and Anticipatory Routing.
• Difficulty of Reproduction: Training Sparse Attention from scratch is extremely difficult, making it unlikely that other teams can easily follow.
• Data and Post-Training: Concrete details about data composition are lacking, and there remains room for improvement in the post-training stage compared to pretraining.

This release suggests that Chinese frontier labs have started to match or in some areas surpass U.S. big tech in the pretraining domain, and indicates that post-training will become the key battleground for next-generation models going forward.