DeepInfer Offline Release: Decode Wide-EP Configuration and Operational Metrics Shared

DeepSeek released DeepInfer, the inference framework for the DeepSeek-V4.1-Flash service, for offline inference scope on October 20, 2026
Supports NVIDIA SM100 GPUs (B200, GB200, etc.) and Huawei Ascend 950 NPUs, while Hopper-generation GPUs are not supported
The release scope excludes the KV cache system for online services, separated deployment of vision encoders, and load balancing features across data parallel replicas

Technical Features and Configuration

Developed based on vLLM, including kernel-level components such as FlashMLA, DeepGEMM, and DeepEP, along with DSpark speculative decoding and SWA bounded replay engine features
SWA bounded replay recalculates only the most recent 128 tokens, reducing decoder prefill computation by nearly half
Applying DSpark variable-length verification yields a 4–7% throughput gain and approximately 30% reduction in verification token count at the operational point (around 60 sequences)

The minimum operational unit is a combination of 64 prefill GPUs and 64 decode GPUs, with the prefill-to-decode GPU ratio set at approximately 1:1
Uses 3FS as the KV cache repository, with approximately 100PiB of SSD space, a 3-day TTL for agent traffic, and a 7-day TTL for chatbot traffic
A single cluster processes 155 trillion tokens per day, with revenue per GPU hour recorded at $38 based on off-peak hours
Licenses are Apache-2.0 for DeepInfer and MIT for model weights, with existing vLLM and SGLang presented as API serving alternatives