AI Briefing#vllm
The latest AI and developer news about #vllm, with the original source and a short summary.
The latest AI and developer news about #vllm, with the original source and a short summary.
The model features 25B active parameters per token and demonstrates strong performance on agentic benchmarks like Terminal-Bench 4.0 and SWE-bench Pro.
The new variants achieve a 6.58% increase in SWE performance and use 20% fewer reasoning tokens while compressing the KV cache by 4x.
The new NIXL connector-based push method reduces Time To First Token by up to 3x and Time Per Output Token by up to 30% compared to pull-based approaches.
Improves SSM kernel performance by up to 1.8x at batch size 256 and speculative decoding throughput by up to 2.1x at batch size 512
The framework now supports serving LLMs via OpenAI, Anthropic, and Ollama protocols simultaneously using a unified .flm file format.
Versions 0.3.9 through 0.5.5 and dev branches are affected, with no fixed release available as of the October 7 advisory.
The new tool ranks GPU kernels by execution time with readable names and supports remote monitoring via SSH or host flags.
The new Apache-2.0 model family, ranging from 0.6B to 27B parameters, achieves state-of-the-art scores on both text and vision boards of the Jev Decision Index.
Added latency SLO-based max concurrency search and agent trace replay in WEKA/OTEL formats.
Pinterest has transitioned production inference to a fine-tuned 4B LLM for user journeys, retaining the legacy clustering system as a fallback, with initial tests showing a 1.1% lift in email click-through rates.
The 5B-parameter model achieves state-of-the-art results on OmniDocBench v1.6 with a score of 96.96 and supports seven distinct parsing families via a unified JSON schema.
Speculative decoding accelerates LLM inference by increasing total FLOPs to utilize idle compute resources, shifting the workload from memory-bound to compute-bound at low batch sizes.
The Aimee project has released vLLM plugins for Qwen 3.8 and Gemma4 models, enabling native memory access that decouples knowledge from model parameters and eliminates the need for retraining.
The MIT-licensed tool bundles Parakeet STT, Kokoro TTS, and SearXNG to simplify local AI access for non-technical users.
New mixed-precision 3-bit weights for Qwen3.8 27B enable a 569,878-token reusable cache on a single AMD R9700, accelerating agent workflows by up to 12x for first-token latency with a 2.9-point MMLU-Pro drop vs MXFP4.
The platform processes approximately 1 trillion tokens daily internally and offers serverless and reserved capacity options.
A security benchmark found that default Ray and vLLM configurations on Kubernetes expose critical control-plane ports to cross-namespace attacks, which standard scanners fail to detect.
The open-source vllm-xtu-moe project achieves up to ~20× faster prefill than CPU-only paths for medium contexts by streaming expert weights from CPU to GPU.
Benchmarking across 29 datasets shows GLM-5.3-Flash achieves statistical parity with the specialized Jev model for intent classification without fine-tuning, though at higher latency and cost.
The new architecture maintains within 3.4% of native token throughput on NVIDIA H100 GPUs while enabling support for out-of-tree accelerators and older hardware.
Released the open-source tool entail to verify and prevent the issue where vLLM's rope_scaling override bug causes GSM8K scores to drop by up to 28%.
Released six open-weight models and a Studio for testing that return the probability of the optimal choice among candidates instead of generating text.
The new TQ method achieves a mean KLD of 0.0282 and 92.4% top-1 accuracy on Qwen 3.8 27B at 4-bit without requiring calibration data.
The new Quail system processes over 1 billion tokens per minute on a single H100 GPU, achieving more than 10x the speed of the vLLM baseline for AI-SQL workloads.