AI Briefing#vllm
The latest AI and developer news about #vllm, with the original source and a short summary.
The latest AI and developer news about #vllm, with the original source and a short summary.
vLLM has introduced a 'hardware-agnostic' layer to resolve compatibility issues with the 'flat model' adopted for maximizing performance on latest GPUs, enabling operation across diverse hardware.
TextCLF has released 'Quant Factory', an open-source tool that enables immediate 4-bit quantization without calibration data.
This analysis compares the features and suitable workloads of major open-source inference stacks such as llama.cpp, vLLM, and SGLang.
GraphSignal, an open-source profiler that allows AI agents to directly read GPU performance data and execute optimization loops, has been released.
Apple researchers have released CapQuiz, a reference-free benchmark for evaluating the information fidelity of video captions.
NVIDIA Dynamo has unveiled a serving architecture that separates vision encoders to reduce the TTFT of multimodal LLMs by up to 93%.
This article introduces vLLM optimization strategies to maximize Gemma 4's SLO-compliant throughput through NVFP4 precision and the separation of interactive and batch pools.
By leveraging Run:ai Streamer and cache optimizations for serving Gemma 4 with vLLM on Amazon EKS, cold start times were reduced from 428 seconds to 226 seconds.
TRL v1.14 introduces Async GRPO functionality that synchronizes LoRA adapters using storage buckets and proxies instead of NCCL.
Cohere released a Megakernel serving engine for the North Mini Code model, achieving up to 1.41x end-to-end inference speedup over vLLM on a single H100 in BF16.
vLLM released detailed benchmark results comparing the throughput and optimal settings of five speculative decoding methods, including Native MTP, EAGLE-3, and DFlash, on AMD MI300X/MI355X GPUs.
Salesforce has released Random Attention, which applies random sampling instead of signal-based methods for KV cache eviction in reasoning models.
NVIDIA expanded its local AI ecosystem at IFA 2026 by unveiling local inference optimization tools and RTX Spark PCs.
NVIDIA has released Model Optimizer, which integrates six optimization techniques including quantization and pruning to support checkpoint generation for inference engines such as vLLM and TensorRT-LLM.
vLLM v0.28.0 adds large-scale performance optimizations and inference capabilities for the Kimi-K3 and DeepSeek V4 models.
The Qwen3-Coder-Next 4bit model, quantized using the calibration-free TQ method, has been released, reducing disk usage to 41.7GB.
vLLM v0.28.0 introduces major performance optimizations and new inference features for Kimi-K3 and DeepSeek V4.
vLLM resolved the non-deterministic output issue for Qwen3.8-Flash-Next at temperature 0 by fixing the sparse-attention kernel.
Tontaube has released TontaubeV1, a 2.9B parameter open-source TTS model specialized for long-form generation and low-latency inference.
It has been raised that LLMs can control host machines by exploiting code execution vulnerabilities in inference engines.
An analysis of the risk that LLMs can exploit inference engine bugs to compromise host servers, illustrated by the misuse of eval() in vLLM.
HBF is discussed as a potential alternative to address DRAM capacity issues in LLM inference.
Toss Securities improved service stability by refining LLM serving monitoring metrics and addressing framework bugs.
PagedAttention applies virtual memory concepts to significantly reduce KV cache memory waste during LLM inference.