vllm releases

Details

Model Runner V2 Optimization: Introduced dual-batch overlap and MTP/EAGLE3 speculative decoding, significantly reducing graph capture time from 12 seconds to 2 seconds and initialization time from 28.9 seconds to 8.2 seconds (on H200).
Fast Start Feature: Introduced a GPU weight-cache daemon supporting --load-format ipc_cache. Applicable to FP4 checkpoints and multi-node TP environments.
NVIDIA Platform: DeepGEMM 2.8.0 (SM120/90), CUDA 13.4 Rubin support, 21% improvement in H20 MoE performance, and NVFP4 KV FMHA acceleration.
AMD Platform: 4~8% improvement in DSv3 QPS and 26% reduction in TPOT for Gemma4 AWQ via ROCm 10.0 and AITER 0.1.21.
CPU Backend: Support for DeepSeek-V4 backend, AMX-FP8 attention optimization for Diamond Rapids, and approximately 6% improvement in Arm LM head performance.

New Models: Support for DeepSeek-V4.1-Flash (MXFP8, SM100), GLM-5.3-Flash, K2-Horizon, Cohere Compass, etc.
Quantization Technologies: FlashInfer CuTeDSL NVFP4 W4A16 replaces Marlin on SM100/103, added 2~7bit CUDA support for AutoRound, and new support for online quantization, W4A16 DSA, NVFP4, and FP8 MoE.
Model-Specific Optimizations: FlashKDA for GLM-5.3-Flash (1.7~3.8x acceleration), fused PLE and FP8 indexer cache for Qwen3.8-Flash-Next, and NVFP4 DeepGEMM and SM100 CUDA AttnRes for Kimi K3.

Scale-out: The --enable-scale-out option has been switched to an opt-in (disabled by default) mode.
GPTQ: Removed g_idx.
YaRN: Changed so that aliases do not readjust max_model_len (reduced from 131k to 32k for TeleChat3).
DCP: Explicit declaration is required when using Distributed Context Parallelism.
Audio: The default resampler has been changed from PyAV to torchaudio.
Deprecated Items Removed: Removed deprecated items from version 0.29 such as VLLM_PREFIX_CACHE_RETENTION_INTERVAL and the all Mamba cache mode.
Known Issue: Backend selection failure (#55780) occurs in ROCm standard/Triton/FlexAttention/TurboQuant. --disable-hybrid-kv-cache-manager is required when using MoRI-IO WRITE with hybrid KV cache.

API: Support for /v1/responses/render, scale-out, reasoning_tokens, and MFU/MBU metrics. Improved OpenAI/Anthropic/Cohere compatibility via Cohere parser integration.
Rust Frontend: Support for TLS, --lora-modules, and HTTP RL synchronization.
Speculative Decoding: Added EAGLE3 Sarvam MLA, Nemotron-H MTP, and NVFP4 DSpark. Adaptive verification is enabled in FLASHINFER_MLA_SPARSE_DSV4.
Watermarking: Adopted Gumbel-max method compatible with speculative decoding.

Security Patches: Blocked validation error response amplification (approx. 5,300x), sparse embedding bounding, GLMGA/Qwen-VL video sampling limits, and applied Cache_salt validation.
Stability: Prevented zero-progress preemption cascades, released NCCL memory and removed dummy forward in sleep mode, and improved CUDA graph fallback and pinned H2D copy.

Details

Model Runner V2 Optimization: Introduced dual-batch overlap and MTP/EAGLE3 speculative decoding, significantly reducing graph capture time from 12 seconds to 2 seconds and initialization time from 28.9 seconds to 8.2 seconds (on H200).
Fast Start Feature: Introduced a GPU weight-cache daemon supporting --load-format ipc_cache. Applicable to FP4 checkpoints and multi-node TP environments.
NVIDIA Platform: DeepGEMM 2.8.0 (SM120/90), CUDA 13.4 Rubin support, 21% improvement in H20 MoE performance, and NVFP4 KV FMHA acceleration.
AMD Platform: 4~8% improvement in DSv3 QPS and 26% reduction in TPOT for Gemma4 AWQ via ROCm 10.0 and AITER 0.1.21.
CPU Backend: Support for DeepSeek-V4 backend, AMX-FP8 attention optimization for Diamond Rapids, and approximately 6% improvement in Arm LM head performance.

New Models: Support for DeepSeek-V4.1-Flash (MXFP8, SM100), GLM-5.3-Flash, K2-Horizon, Cohere Compass, etc.
Quantization Technologies: FlashInfer CuTeDSL NVFP4 W4A16 replaces Marlin on SM100/103, added 2~7bit CUDA support for AutoRound, and new support for online quantization, W4A16 DSA, NVFP4, and FP8 MoE.
Model-Specific Optimizations: FlashKDA for GLM-5.3-Flash (1.7~3.8x acceleration), fused PLE and FP8 indexer cache for Qwen3.8-Flash-Next, and NVFP4 DeepGEMM and SM100 CUDA AttnRes for Kimi K3.

Scale-out: The --enable-scale-out option has been switched to an opt-in (disabled by default) mode.
GPTQ: Removed g_idx.
YaRN: Changed so that aliases do not readjust max_model_len (reduced from 131k to 32k for TeleChat3).
DCP: Explicit declaration is required when using Distributed Context Parallelism.
Audio: The default resampler has been changed from PyAV to torchaudio.
Deprecated Items Removed: Removed deprecated items from version 0.29 such as VLLM_PREFIX_CACHE_RETENTION_INTERVAL and the all Mamba cache mode.
Known Issue: Backend selection failure (#55780) occurs in ROCm standard/Triton/FlexAttention/TurboQuant. --disable-hybrid-kv-cache-manager is required when using MoRI-IO WRITE with hybrid KV cache.

API: Support for /v1/responses/render, scale-out, reasoning_tokens, and MFU/MBU metrics. Improved OpenAI/Anthropic/Cohere compatibility via Cohere parser integration.
Rust Frontend: Support for TLS, --lora-modules, and HTTP RL synchronization.
Speculative Decoding: Added EAGLE3 Sarvam MLA, Nemotron-H MTP, and NVFP4 DSpark. Adaptive verification is enabled in FLASHINFER_MLA_SPARSE_DSV4.
Watermarking: Adopted Gumbel-max method compatible with speculative decoding.

Security Patches: Blocked validation error response amplification (approx. 5,300x), sparse embedding bounding, GLMGA/Qwen-VL video sampling limits, and applied Cache_salt validation.
Stability: Prevented zero-progress preemption cascades, released NCCL memory and removed dummy forward in sleep mode, and improved CUDA graph fallback and pinned H2D copy.

vllm