vllm

Key Changes

🚀 Major Features and Architectural Changes

• Model Runner V2 (MRV2) Default Adoption: MRV2 is now set as the default execution engine for all models. MRV1 is no longer the default, except for some ROCm models and unsupported features.
  • KV Cache Auto-Sizing: Automatically optimizes KV cache size via CUDA graph memory profiling.
  • Improved Memory Efficiency: Introduces batch-sharded sampling to reduce per-step logits memory by 1/TP (Tensor Parallelism).
  • Additional Features: Support for prompt embeds, extract_hidden_states speculation, padded FULL cudagraph dispatch for uniform decoding, and skipping DP synchronization before EAGLE/MTP draft prefill.

🆕 New Model Support

• Hy4-preview: Tencent's 770B parameter (49B active) MoE model, supporting Gated DeepSeek Sparse Attention and native MTP.
• Qwen3.8-Flash-Next: Supports BF16, FP8, NVFP4 formats and MTP.
• Other Models: GraniteSWA, GraniteMoeSWA, NemotronH_Omni_Reasoning_V3 (MTP support), Kimi K3 NVFP4 checkpoints.

⚡ Performance Optimizations

• Kimi-K3 and DeepSeek V4 Optimizations:
  • Kimi-K3: Approximately 5% reduction in end-to-end (E2E) latency via MXFP4 top-k finalization fusion, 6.6–7.6x kernel speedup by integrating Mamba metadata preparation into a single Triton launch, and 12.9–25.2% improvement in eh_proj kernel speed via Hopper low-latency GEMM tuning (including SM100 dispatch).
  • DeepSeek V4: Fuses shared experts into MegaMoE, reintroduces adaptive top-k width, native SwiGLU clamp kernel for Humming MoE, and support for selective FlashInfer moe_ep expert backend.
• Speculative Decoding:
  • Provides per-request acceptance stats via the --per-request-spec-decode-metrics flag in OpenAI API responses.
  • Extends adaptive verification for logprobs.
  • SM100 Sparse MLA support for GLM-5.2 and DeepSeek V4 (SM90).
  • DSpark drafts support for Qwen3-Omni.