DeepSeek-V4, 1M-token context

DeepSeek released V4. The core point is not simply competing on benchmarks, but a long-context design that lets agents actually complete long tasks from start to finish.

• DeepSeek-V4-Pro: 1.6T total parameters, 49B active
• DeepSeek-V4-Flash: 284B total parameters, 13B active
• Both models support a 1M-token context window.

At 1M tokens, V4-Pro's single-token inference FLOPs are about 27% and KV cache about 10% of V3.2's levels, while V4-Flash's FLOPs are about 10% and KV cache about 7%. Compared to a standard GQA 8-head, BF16 baseline, KV cache is reduced to roughly 2%.

The efficiency comes from hybrid attention.

• CSA (Compressed Sparse Attention): compresses KV by 4x and selects only the top-k among the compressed blocks.
• HCA (Heavily Compressed Attention): compresses KV by 128x and then applies dense attention.
• In V4-Pro's 61-layer stack, the first 2 layers are HCA, followed by alternating CSA and HCA.

Post-training for agents has also changed. In conversations that include tool calls, reasoning is preserved up to user message boundaries so that the state of long tasks isn't interrupted, and a |DSML| token along with an XML-based tool-call format were introduced to reduce the escaping errors that often occurred with JSON-in-string. For RL rollout infrastructure, DSec is used, which bundles function calls, containers, Firecracker microVM, and QEMU VM into a single Python SDK.

Performance is not SOTA across the board, but it is strong on agentic tasks. It scored Terminal Bench 2.0 67.9, SWE Verified 80.6, MCPAtlas Public 73.6, and Toolathlon 51.8, and on an internal R&D coding benchmark it achieved a 67% pass rate, ahead of Sonnet 4.5's 47%. On the long-context retrieval test MRCR 8-needle, it maintains above 0.82 up to 256K and records 0.59 at 1M.