AI BriefingDFlash and Spec V2 Decoding Technology
Key point
DFlash technology, developed jointly by Z Lab and Modal, achieved LLM inference performance far superior to existing MTP methods.
Details
Z Lab, SGLang, and Modal unveiled technology combining the DFlash and Spec V2 engines to achieve state-of-the-art latency in LLM inference serving. In particular, in benchmarks targeting the Qwen 3.5 397B-A17B model, DFlash recorded 4.3x higher throughput than the baseline and 1.5x higher throughput than MTP (Multi-Token Prediction) on the HumanEval dataset.
Existing Speculative Decoding methods such as EAGLE and MTP relied on a sequential autoregressive approach in which the draft model generates tokens one at a time, resulting in low hardware efficiency. DFlash, on the other hand, uses a lightweight Block Diffusion draft model to generate an entire block of tokens in parallel at once, in a manner optimized for GPU/TPU architectures.
The core innovation of DFlash lies in its KV Injection strategy. By extracting hidden representations from the target LLM's context tokens and injecting them into the draft model, the draft model is designed to best understand the target model's context. This resolves the low acceptance length problem and dramatically increases inference speed.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
3.85x Inference Speedup via Diffusion Draft Generation and AR Verification Based on GLM-OCR
The Efficient Frontier of LLM Inference (6-minute read)vLLM v0.28.0 Released with Major Optimizations for Kimi-K3 and DeepSeek V4