AI Briefing#vllm
The latest AI and developer news about #vllm, with the original source and a short summary.
The latest AI and developer news about #vllm, with the original source and a short summary.
Qwen3.6-27B was run on 2x 3090 GPUs at 118 tokens per second using a combination of vLLM Docker with INT4 and MTP.
A single H100 80GB benchmark confirmed the advantage of Gemma 4 E2B and Qwen 3.6 A3B FP8.
In a single H100 GPU benchmark, Gemma 4 E2B-it delivered the best results in both throughput and TTFT.
A PR supporting Cohere MoE models has been submitted to vLLM.
A parser error in vLLM was found to cause malfunctioning Tool Calling functionality in Qwen models.
vLLM Recipes has been revamped to support hardware-specific serving configs and command generation.
Built a chat UI for Gemma-4-E4B using a Qwen-series model in about 8 minutes.
A single checkpoint switches among 8 presets, delivering up to 10.7x faster performance.
This summarizes the major technical issues and solutions that arise during fine-tuning and deployment of the Gemma-4 model.
Ran Qwen3.6 35B 4bit for multi-user use via vLLM Docker on 2x RTX 3090.
A benchmark, safety, and tensor analysis comparing Qwen 3/3.5 with HauhauCS, Heretic, and Huihui was released.
In an apples-to-apples FP8 comparison, the advantage between Ironwood and B200 reversed depending on the model.
On A100 40GB, MIG significantly lowered protected-tenant p95 compared to shared vLLM.
Qwen3.6 is the latest Qwen model family with strengthened agentic coding and thinking-context preservation.
A benchmark running MiniMax M2.7 AWQ on 8x MI50 32GB hit a peak of 64 tok/s.
Trained Qwen2.5-0.5B-Instruct on a Reddit summarization task using GRPO.
Shared real-world benchmarks and issues from running Qwen3.5-27B on the Arc B70.
A high-resolution CLI profiler for vLLM that catches batching, KV, and cache issues.
At the Seoul meetup, participants confirmed the trend of vLLM growing into standard infrastructure for LLM serving.
The nondeterminism of LLM inference is less about GPU concurrency and more about request scheduling.
For 96GB VRAM setups, Qwen 3.5 122B and 27B have emerged as the practical choices.
Verified Qwen3.5-122B running at 198 tok/s on 2x RTX PRO 6000 Blackwell.
TriAttention compresses the KV cache by 10.7x and improves throughput by 2.5x through trigonometric-based compression without loss of accuracy.