AI Briefing#vllm
The latest AI and developer news about #vllm, with the original source and a short summary.
The latest AI and developer news about #vllm, with the original source and a short summary.
In hybrid model serving, it increased prefill TPM by up to 3.14x and reduced p90 TTFT by up to 88%.
The release expands hardware support to include two-GPU setups and achieves 75.5 tok/s on a single RTX PRO 6000 card.
A technical guide details serving the 753B parameter GLM-5.3 NVFP4 model across two NVIDIA DGX Stations using vLLM, achieving a peak throughput of 607.8 tokens/sec.
The framework supports weight and activation quantization for LLMs, text-to-image, and text-to-video models while maintaining inference speed.
PyTorch's Helion now uses compile-time heuristics to select fast kernel configurations without autotuning, achieving 1.18x speedup over Flash Linear Attention and 1.88x over torch.compile for matmuls on H100.
vLLM Router provides routing functionality that distributes inference requests while considering sessions and caches.
MACaQ quantizes a 122B-parameter model in 64 minutes using 32.5 GB of device memory, supporting text, image, video, and audio models across llama.cpp, vLLM, Diffusers, and PyTorch.
The MIT-licensed tool analyzes KV-cache events from vLLM and SGLang to pinpoint why requests miss the prompt cache, achieving 32/32 agreement with Anthropic's diagnostics.
Prefixwise v0.10.0 identifies and fixes LLM prompt cache misses, realizing 99.9% of predicted savings on Claude and 97.5% on vLLM.
Removing unnecessary draft-token reservations allows larger prefill chunks and more efficient CUDA graph execution, raising output throughput from 1,373 to 3,390 tok/s.
Modular has released a free 57-page handbook covering LLM inference metrics, optimization, and operational costs.
Concurrency tests on H200 GPUs show the model retains its efficiency edge, achieving 985 correct answers per hour at 128 users versus 616 for the base model.
The team combined mixed-precision quantization, DFlash2 speculative decoding, and custom CUDA kernels to reduce latency and improve throughput on a single GPU.
New Docker image bundles vLLM fork and ROCm 10.1 for Strix Halo, achieving 40-43 tok/s decode for Qwen3.8-Flash-Next.
The open-source plugin targets gfx1151 hardware to optimize MoE GEMMs and attention layers, reducing 256k prompt prefill time from over two hours to four minutes.
Enabling the vLLM 0.31 server-side constrained decoding flag improved tool-call schema accuracy from 92% to 100% for Kimi K2.6 and DeepSeek models.
Unclosed connections after user cancellation or retries cause vLLM to generate up to 175 wasted tokens per task, reducing on-time answer rates to 11%.
The study benchmarks KV cache limits and conversation state management strategies for agent model switching.
The library supports NVIDIA Turing and later architectures, achieving significant speedups over vLLM Marlin and DeepGEMM across dense and MoE workloads.
The open-source Kubernetes-native stack supports vLLM and SGLang across heterogeneous hardware, claiming up to 3x higher output throughput and 2x faster TTFT.
A new Helion linear backend for vLLM uses AOT autotuning to outperform CUTLASS and DeepGEMM baselines on NVIDIA Hopper GPUs, with >10% throughput gains on some workloads.
The new vllm-torchtpu plugin allows standard PyTorch models to run on TPU v7x with no code changes, using custom JAX/Pallas kernels for attention and quantization.
While performance is similar under default settings, the optimal engine varies depending on output length and prefix reuse.
The Apache 2.0-licensed model features 23B active parameters, a 1M token context window, and day-0 support for vLLM, SGLang, and Transformers.