vllm

Key Features and Performance Improvements

• Kimi-K3 Optimization: Decode Context Parallel (DCP) support, FlashKDA decode/prefill kernel fusion, SiTU activation for MegaMoE, and GEMM-RS introduction for sequence parallelism. All-gather fusion delivers 1.5-3x kernel speedup, adaptive speculative token budget improves DSpark TTFT by about 60%, and selective shared expert sharding saves about 17 GiB of memory per GPU. Added ROCm V2 model runner support.
• DeepSeek V4 End-to-End Support: Sparse MLA support for normal decode, MTP, and DSpark speculative decoding. AMD Quark NVFP4 support, reasoning-effort prompt mapping, sparse top-k metadata kernel optimization, reduced Eager CUDA graph region, and ROCm enablement based on gfx11/gfx950.
• Speculative Decoding Progress: DFlash2 with local convolution and candidate selector, DSpark with confidence-based verification scheduling, and automatic activation of asynchronous scheduling for draft models.
• Model Runner V2 Maturity Improvements: E/P/D disaggregation, weight offloading, multi-layer MTP KV cache support, encoder CUDA graphs, decoder token-wise pooling, Transformers pooling model support, attention-free model support, and thinking_token_budget support.
• Tiered KV Cache Offloading: Disk offloading support, out-of-tree secondary tier manager via module_path, handling of partial secondary tier load results, tiering metrics, and standardized CPU layout for parallelism-agnostic offloading.
• Rust Frontend and gRPC: Standalone renderer, multimodal image inference support via gRPC.