vllm

🚨 Key Changes and Compatibility (Breaking Changes)

• Python 3.10 Support Dropped: Requires Python 3.11 or higher.
• Dependency Updates: Requires transformers>=5.16.1 and tokenizers>=0.23.2.
• Security Enhancements: CORS defaults to none (blocked by default). Watermark is enabled by default.
• Feature Removals: Removed Cascade attention, torch.compile SP/AsyncTP, and mamba_cache_mode=all.
• API Changes:
  • max_tokens redefined as the output limit.
  • Use reasoning_text in the Responses API.
  • logprobs changed to integer IDs.
  • parallel_tool_calls enforced.
  • Added /v1/systemone endpoint.
• Default Changes: Shared-memory arena enabled by default, Model Runner V2 used by default on ROCm.

🚀 Performance Improvements

• Low-Precision KV Cache: Optimizations for low-precision caches like NVFP4 and FP8 improved throughput by +2.3~5.5% on GLM-5.x and output by +31% on fp8_ds_mla.
• HiSparse Caching: Achieved +58% throughput.
• Startup Speed Improvements: Introduced vllm download-kernels and --preload-autotune; setting --api-server-count>1 increased short req/s by 43~85%.
• Hardware-Specific Optimizations:
  • NVIDIA SM100: FlashMLA TTFT improved by 1.16~1.18x.
  • AMD ROCm: Density improved by ~2x with ultraquant_4bit, Kimi-K3 TTFT reduced by -20%.
  • Intel XPU: Arc B70 FP8 GEMM improved by 2.0~2.73x.
  • CPU (s390x): Sampler performance improved by 58x.
• Other Optimizations:
  • MoE weighted-sum tuning improved by 1.17~2.37x.
  • Detokenization CPU usage reduced by 69%.
  • NCCL symmetric-memory MoE reduce-scatter latency reduced by 9.6~16%.
  • DeepEPv2 hybrid NVLink+RDMA enabled by default.

🆕 New Features & Models

• New Model Support: Added EmbeddingGemma 2, MiniCPM-V 4.7, Nemotron 3.5 ASR, GLM-5.3/Flash, DeepSeek-V4/V4.1, Qwen3.8-Flash-Next, Kimi-K3, MiniMax-M3, etc.
• Model-Specific Optimizations:
  • GLM-5.3/Flash: Sparse MLA index reuse, fused draft decode, long-context indexer TP shard applied (TTFT reduced by up to 25% for 1M context).
  • DeepSeek-V4/V4.1: Decoder-side SWA bounded replay (+3.6%), encoder CUDA graph packing (TTFT reduced by 11%).
  • Qwen3.8-Flash-Next: QSA improved by 1.45~1.68x, SM100 decode GEMM improved by 2.7x.
  • Kimi-K3: Vision preprocess improved by 10~20x.
  • MiniMax-M3: MSA prefill improved by 26~37%.
• Model Runner V2: Enabled by default on ROCm, supports GPU n-gram speculator and Elastic EP.
• Sleep/RL Features: Optimized memory release for workspace scratch and KV-init states.
• Rust Frontend: Separated qwen3_xml and qwen3_json.