AI Briefing#vllm
The latest AI and developer news about #vllm, with the original source and a short summary.
The latest AI and developer news about #vllm, with the original source and a short summary.
NVIDIA has released a quantized version of Alibaba's Qwen3.6-35B model using the NVFP4 data type.
A native HIP W4A16 kernel that optimizes performance on AMD GPUs has been added to vLLM.
TRL library added a Delta Weight Sync feature that transfers only the changed weights, dramatically reducing bandwidth consumption during RL training.
LG AI Research leveraged vLLM metrics to repurpose idle GPUs for research jobs, maximizing infrastructure efficiency.
Harbor v0.4.19 has been released, supporting agent execution based on local inference backends and automatic injection of web search tools.
Eagle 3.1 was released through a collaboration between the EAGLE, vLLM, and TorchSpec teams.
A security vulnerability has been discovered in which insufficient Host header handling in Starlette can allow authentication to be bypassed in major AI infrastructure such as vLLM and LiteLLM.
A tool has been released that measures a GPU's actual effective utilization and its upper bound in real time.
Marlin, a 2B open-source VLM based on Qwen3.5-VL, has been released.
Amazon extended the Chinchilla scaling law to present a framework that speeds up LLMs without losing accuracy.
A template error in the Qwen3.5 Vision model has been fixed in vLLM.
vLLM concluded that FP8 KV-cache is the default choice, while TurboQuant's higher-bit options offer little real benefit.
Borealis has released an open-source training recipe and a 5B model for audio LLMs.
MiniCPM has released MiniCPM-V 4.6, unveiling a 1.3B vision-language model for mobile use.
Automated Agentic AI operations using Amazon EKS Auto Mode, Bifrost, and Langfuse.
Fixed vLLM V1's logprob, cache, and update discrepancies to align with the V0 trajectory.
vLLM Factory, an open-source tool that efficiently serves encoder models such as GLiNER and ColBERT via a vLLM plugin, has been released.
vLLM has resolved the unimplemented TurboQuant error affecting Qwen 3.5+ models caused by an issue with Mamba layers.
In mixed traffic, class-aware vLLM routing showed the best balance.
On 4x 5060 Ti, vLLM recorded pp 1444.9 tok/s and tg 47.4 tok/s.
A vLLM PR has added support code for Mistral-Medium 3.5 128B.
On 2x3090, a 250W power limit emerged as the efficiency sweet spot for Qwen3.6-27B.
The proliferation of open-weight models is shaking the monopoly-moat logic of US AI.