AI Briefing#llm-serving
The latest AI and developer news about #llm-serving, with the original source and a short summary.
The latest AI and developer news about #llm-serving, with the original source and a short summary.
In hybrid model serving, it increased prefill TPM by up to 3.14x and reduced p90 TTFT by up to 88%.
The open-source engine reduces batch-8 decode latency to 24.2 ms by fusing layers into single NKI kernels to minimize HBM traffic and compiler overhead.
The system prioritizes full-attention KV cache admission to reduce SSD wear without sacrificing cache hit rates.
DeepSeek has released the offline inference code for DeepInfer, its inference framework for V4.1-Flash.
The release supports NVIDIA SM100 and Huawei Ascend 950 hardware, featuring optimized kernels and asynchronous scheduling for offline inference.
Pie is an open-source LLM serving system where user programs (Inferlets) directly control the KV cache and generation loops, reducing latency by up to 15% for agent workloads like ReACT.
The framework now supports serving LLMs via OpenAI, Anthropic, and Ollama protocols simultaneously using a unified .flm file format.
New mixed-precision 3-bit weights for Qwen3.8 27B enable a 569,878-token reusable cache on a single AMD R9700, accelerating agent workflows by up to 12x for first-token latency with a 2.9-point MMLU-Pro drop vs MXFP4.
The platform processes approximately 1 trillion tokens daily internally and offers serverless and reserved capacity options.
An open-source repository has been released for serving the 320B MoE model GLM-5.3-Flash on Kaggle's free TPU v5e-8.
By leveraging Run:ai Streamer and cache optimizations for serving Gemma 4 with vLLM on Amazon EKS, cold start times were reduced from 428 seconds to 226 seconds.
Cohere released a Megakernel serving engine for the North Mini Code model, achieving up to 1.41x end-to-end inference speedup over vLLM on a single H100 in BF16.
vLLM released detailed benchmark results comparing the throughput and optimal settings of five speculative decoding methods, including Native MTP, EAGLE-3, and DFlash, on AMD MI300X/MI355X GPUs.
Paddock, a Rust/C++ inference engine featuring custom CUDA kernels, has been open-sourced under the MIT/Apache-2.0 license.
Cerebras has started serving the Qwen 3.8 27B model on its public endpoint at a speed of approximately 1,500 tokens per second.
FreeToken proposes a system for efficiently serving frontier-grade MoE models on hardware comparable to gaming PCs.
FreeToken has released an edge-native serving engine that enables interactive-speed execution of frontier MoE models on consumer hardware.
Toss Securities improved service stability by refining LLM serving monitoring metrics and addressing framework bugs.
LMSys has released optimization strategies to maximize DeepSeek-V4-Pro serving performance on H20 GPUs.
An adaptive MTP feature that dynamically adjusts MTP depth based on output complexity has been proposed for llama.cpp.
The PyTorch Foundation revealed the latest achievements and roadmaps for 6 projects, including PyTorch and vLLM.
Instead of using hosted APIs, Netflix built its own LLM serving infrastructure to directly operate the entire stack from model deployment to inference.
The patch fixes unnecessary regex recompilation occurring when using sm-tensor in llama.cpp, improving decode performance.
ELDR routing, which accounts for expert locality in MoE models, cuts decode latency by up to 13.9%.