vllm

🚀 Key New Features and Performance Improvements

• DeepSeek-V4.1-Flash Support: Maximizes performance by enabling FlashMLA and NVFP4 by default.
• Fast Restart and Memory Management: Significantly improves service restart speed by supporting GPU memory caching via vllm preload and CRIU-based snapshot features.
• Model Runner V2 Expansion: Adds LiLiCorr drafter and DFlash asynchronous scheduling, improving large-scale serving performance through the MoonEP backend and Prefill CP.
• New Model Support: Supports DiffusionGemma, MiMo V2, Cohere2MoE, GLM-5.2/5.3, Granite 4.2, and others.
• Model-Specific Optimizations:
  • DeepSeek V4: Applies FlashInfer sparse MLA and fused MoE.
  • GLM-5.3-Flash: Accelerates metadata processing by 1.6-4.8x and saves 3GiB of memory using the SM90/120 sparse backend.
  • Qwen3.8-Flash-Next: Supports FP8 KV/MoE and reduces DGX Spark weight loading time by 25 seconds.
  • Kimi K3/MiniMax-M3: Accelerates RoPE operations by 29x and Conv3d operations by 62x.
  • DiffusionGemma: Accelerates logprob reading speed by 25%.
• Scheduler and Caching Improvements:
  • Introduces --max-num-active-seqs and adaptive --long-prefill-token-threshold.
  • Replaces skipped_waiting with a KV-holding queue and resolves the throughput cliff issue occurring when max_num_seqs is not a multiple of 8.
  • Supports Mamba shared checkpoint in prefix caching and eliminates GPU-CPU sync via batched Mamba2 prefill state save.
• Platform-Specific Optimizations:
  • NVIDIA: FlashInfer 0.7.0.post1, DeepGEMM SM120 fix, PyTorch 2.13 QuTLASS support.
  • AMD ROCm: AITER v0.1.23, Triton 3.8 support, 13x reduction in decode latency via Hy4 torch.compile.
  • Intel XPU: PyTorch 2.14 support, XPU graphs enabled by default, 55% decode acceleration via fused QK RMSNorm+RoPE+gate.
  • CPU: Up to 25% acceleration in Arm paged attention, FP8 W8A8/INT4 support, Ubuntu 22.04 wheels provided.

⚠️ Breaking Changes and Removed Features

• Tokenizer Mode Change: The tokenizer_mode="slow" option has been removed.
• Quantization Change: The fp8 option has been changed to fp8_per_tensor.
• AllSpark INT8 W8A16 GEMM Removal: Support for this backend has been discontinued.
• JIT Warmup Disabled: JIT warmup is disabled by default when using --enforce-eager.
• XPU Graph Enabled by Default: XPU graphs are enabled by default in Intel XPU environments.
• Environment Variable Removal:
  • VLLM_PLE_CPU_OFFLOAD removed (use --engram-config instead).
  • VLLM_XPU_ENABLE_XPU_GRAPH removed (XPU graph is ON by default).
• API Change: return_assistant_tokens_mask removed.
• Mamba Prefix Cache Option Change: --enable-mamba-fine-grained-prefix-cache has been changed to --enable-mamba-shared-prefix-checkpoint.

🔒 Security Enhancements

• Multimodal Parameter Blocking: Per-request multimodal parameters (mm_processor_kwargs) are blocked by default unless the --trust-request-mm-kwargs option is used.
• LoRA Hash Collision Prevention: Improved to prevent LoRA-related hash collision issues.
• Prefix Cache Hash Strengthening: Strengthened the hashing logic for prefix caching to enhance security.

🛠️ Other Major Changes

• Structured Output: Supports XGrammar 0.2.7 and Granite Streaming Parser.
• API Compatibility: Improved OpenAI/Anthropic/Harmony compatibility, supporting reasoning token count and batched completions.
• Rust Frontend: Includes --hf-overrides, gRPC sampling masks, vllm-proto 0.3.0, and vllm-rs in CUDA images.
• Sleep/RL Support: Releases only KV cache while preserving frozen weights via release_kv_cache_memory(), and supports NCCL M2N weight transfer.
• Batch Invariance: Uses breakable CUDA graphs by default without torch.compile under the VLLM_BATCH_INVARIANT environment variable.