AI Briefing#model-serving
The latest AI and developer news about #model-serving, with the original source and a short summary.
The latest AI and developer news about #model-serving, with the original source and a short summary.
A technical guide details serving the 753B parameter GLM-5.3 NVFP4 model across two NVIDIA DGX Stations using vLLM, achieving a peak throughput of 607.8 tokens/sec.
The Reshard operation announced by NVIDIA reduces weight transfer time in disaggregated RL environments by up to 5.7x.
Amazon engineers swapped Claude Sonnet 4.6 for the Strands Decider E4B model in their internal ticketing system, reducing per-ticket costs from $0.0222 to approximately $0.0005.
Built on llama.cpp, the tool automatically allocates model weights to VRAM, RAM, or flash storage to maximize prefill and decode speeds on heterogeneous hardware.
The open-source tool rewrites its own prompts and compiled core through four verification gates, achieving a 4.3-second mid-conversation swap while maintaining 87.9% prompt cache retention.
The new vllm-torchtpu plugin allows standard PyTorch models to run on TPU v7x with no code changes, using custom JAX/Pallas kernels for attention and quantization.
The new backend enables eager execution and native distributed APIs in Transformers, Diffusers, and Kernels, replacing the previous Optimum Neuron workflow.
The hybrid approach uses three tiny MLPs on a frozen Gemma 4 backbone to decide whether to speak, defer, or wait, achieving up to 99.0% start-turn accuracy on local benchmarks.
The new TQ method achieves a mean KLD of 0.0282 and 92.4% top-1 accuracy on Qwen 3.8 27B at 4-bit without requiring calibration data.
This analysis compares the features and suitable workloads of major open-source inference stacks such as llama.cpp, vLLM, and SGLang.
Intel has released OpenVINO 2026.4, adding support for the latest models such as Gemma 4 and Qwen 3.5/3.6, along with Multi-Token Prediction inference acceleration.
Reef has released a recipe for deploying OpenClaw-RL weight updates after verification.
Human-Agent-Society has released 'Reef', an open-source infrastructure that updates model weights and harnesses in real time using user feedback during agent serving.
vLLM v0.28.0 introduces major performance optimizations and new inference features for Kimi-K3 and DeepSeek V4.
Optimized Qwen3-TTS to achieve sub-50ms latency and 10 RPS throughput on a single H100.
LLMRouter and xRouteBench, featuring over 16 router libraries and benchmarks for LLM cost optimization, have been released.
Qwen has released the open model Qwen3.8 with a total of 2.4T parameters and 95B active parameters.
Anthropic is investigating increased errors and degraded performance across several Claude models.
WARP has implemented local inference for the 2.78-trillion-parameter Kimi K3 on a 64GB MacBook.
SIE unifies serving of embeddings, reranking, OCR, and LLMs through a single OpenAI-compatible API.
NVIDIA has released Magpie TTS, an open-weight model supporting 12 languages.
Muse Glimmer has been released as a 30B open-weight agent model capable of running on 24GB-class memory.
DeepSeek has released the official version of V4-Flash, which supports a 1 million token context window.
Hugging Face has added Baseten as an Inference Provider to support serverless model inference.