AI Briefing#llm-serving
The latest AI and developer news about #llm-serving, with the original source and a short summary.
The latest AI and developer news about #llm-serving, with the original source and a short summary.
SGLang is introducing executable skills for agents (SKILL.md) to automate complex LLM and Diffusion workflows.
With Hugging Face Jobs infrastructure, you can build an OpenAI-compatible vLLM endpoint with just a single command.
The Shard project dramatically improved distributed inference speed for large-scale models by leveraging speculative decoding and other techniques in a WAN environment.
Xiaomi has unveiled MiMo V2.5Pro, a model that delivers inference speeds of up to 3,000 tokens per second.
This covers a case study on introducing MLXP optimization technology to improve LLM serving efficiency in Kubernetes environments.
Apple has released MLX LM Server, which supports Continuous Batching and distributed inference.
Luce Spark has been released, learning real-time routing patterns to efficiently keep MoE experts resident on the GPU.
OpenBMB released BitCPM4-CANN 1B, 3B, and 8B on Hugging Face.
vLLM concluded that FP8 KV-cache is the default choice, while TurboQuant's higher-bit options offer little real benefit.
Ollama has switched from a GGML-based build to direct llama.cpp support.
Ollama has added context length, parameter count, and quota usage information to model cards.
A memory leak causing data exposure and a remote code execution (RCE) risk via Windows Update have been discovered in Ollama.
Atlas, a high-performance inference engine built on Rust and CUDA, has been released as open source.
FastDMS has been released, optimizing the KV-cache compression technology DMS to achieve less memory usage and faster speed than vLLM.
SMG separated serving CPU work into a Rust gateway to eliminate the Python GIL bottleneck.
Per-GPU KV cache locality determined LLM serving cost and TTFT.
In a 2 x RTX Pro 6000 environment, Qwen3.6-27B delivered the best performance, while Qwen3.6-35B-A3B showed higher tps.
A Metal kernel implementation combining INT3 model compression with INT2 KV cache has been released.
By leveraging the small KVCache of hybrid-attention models, prefill is moved outside the data center.
On DGX Spark, Qwen3.6 performs similarly to Qwen3.5, but MTP significantly increases throughput.
At the Seoul meetup, participants confirmed the trend of vLLM growing into standard infrastructure for LLM serving.
A 4-chiplet AI SoC implements a full-chip mesh with 16Gbps UCIe-Advanced.
OVHcloud has joined as an Inference Provider for Hugging Face, supporting serverless AI model services.
Hugging Face has added Public AI as a new inference provider to support public AI models.