AI Briefing#throughput
The latest AI and developer news about #throughput, with the original source and a short summary.
The latest AI and developer news about #throughput, with the original source and a short summary.
Removing unnecessary draft-token reservations allows larger prefill chunks and more efficient CUDA graph execution, raising output throughput from 1,373 to 3,390 tok/s.
AI Gateway has launched a unified Fast Mode in beta that provides lower latency and higher throughput on a per-model basis.
AI Gateway has added a `sort` option that sorts providers by cost, TTFT, and TPS.
Per-GPU KV cache locality determined LLM serving cost and TTFT.
On 2 nodes with 32 AMD MI50s, Kimi K2.6 int4 achieved 9.7 tok/s output and 263 tok/s input.
Qwen3.6-27B-INT4 achieved 256k context and 105~108 tps on a single RTX 5090.
The author reported running Qwen3.6-27B on a single RTX 5090 with vLLM 0.19.1rc1, achieving about 80 tokens/sec at a 218k context.
With llama.cpp ngram-mod speculative decoding, Qwen-3.6-27B speed rose to 136.75 t/s.
Kimi Linear makes cross-datacenter Prefill/Decode disaggregation practical, lowering costs.
On DGX Spark, Qwen3.6 performs similarly to Qwen3.5, but MTP significantly increases throughput.
A diffusion LLM has achieved both AR-level quality and parallel generation speed together.
Vercel AI Gateway now provides real-time throughput and latency metrics for hundreds of models.