DeepSeek v4

DeepSeek has released the DeepSeek-V4 series, which supports 1M-token context. There are two models: V4-Pro (1.6T total, 49B active) and V4-Flash (284B total, 13B active), both downloadable from Hugging Face and ModelScope.

The core architecture is a hybrid attention combining CSA and HCA. At 1 million tokens, it reduces single-token inference FLOPs to 27% compared to DeepSeek-V3.2, and cuts KV cache usage to about 10%. It also adds mHC connections and the Muon optimizer to improve signal propagation stability and convergence speed.

Training was conducted through pretraining on 32T+ tokens followed by two-stage post-training.

• Stage 1: Individually training domain-specific experts via SFT + RL (GRPO)
• Stage 2: Integrating expert capabilities into a single model via on-policy distillation

There are three inference modes: Non-Think, Think High, and Think Max. Performance increases with higher modes, and V4-Pro-Max showed strong performance in coding and agent tasks.

On benchmarks, it achieved LiveCodeBench 93.5, SWE Verified 80.6, and Codeforces 3206, and the Base model also surpassed the V3.2 series with MMLU 90.1, HumanEval 76.8, and LongBench-V2 51.5. For deployment precision, the Base model uses FP8 Mixed and the Instruct model uses FP4 + FP8 Mixed; instead of a chat template, an OpenAI-compatible encoding/parsing script is provided. The license is MIT.