AI Briefing#inference
The latest AI and developer news about #inference, with the original source and a short summary.
The latest AI and developer news about #inference, with the original source and a short summary.
It's a CLI tool that analyzes your hardware specs to recommend and run the local LLM that delivers the best performance.
llama.cpp b9200 includes an optimization that improves MTP prompt processing speed.
vLLM concluded that FP8 KV-cache is the default choice, while TurboQuant's higher-bit options offer little real benefit.
Furiosa revealed RNGD's transition to mass production and its inference strategy at RENEGADE Summit 2026.
llama.cpp's server and webui now support the continue generation feature for reasoning models.
AI agent demand is exposing the limits of GPU- and HBM-centric infrastructure.
Modal reduced serverless GPU startup to around 50 seconds using buffers and checkpoint restore.
TabPFN-3 has been released, supporting inference on up to 1 million rows on a single H100.
A local 35B model handled about half of daily tasks faster than the cloud.
The llama.cpp b9095 update enables tensor parallel processing on Blackwell PCIe GPUs without NCCL.
Google released an MTP drafter for Gemma 4 that boosts inference speed by up to 3x.
Analysis shows LLM weights in BF16 carry only about 10.6 bits of actual information out of 16 bits.
Google has released MTP drafters for Gemma 4, boosting inference speed by up to 3x.
vLLM has resolved the unimplemented TurboQuant error affecting Qwen 3.5+ models caused by an issue with Mamba layers.
Random cross-layer attention enables layer-wise KV cache sharing.
Analyzes memory and compute scaling in LLMs along with batching strategies for efficient inference.
LLM inference splits into prefill and decode, and the KV cache determines speed.
Hummingbird+ ran Qwen3-30B-A3B Q4 at 18 t/s on a board with 24GB and a mass-production price of about $150.
On Windows 10 with native vLLM, Qwen3.6-27B hit 72 tok/s.
Unsloth fixed the Mistral Medium 3.5 inference bug and updated the GGUF.
Z.AI identified the cause of GLM-5's intermittent output anomalies as an infrastructure issue.
During large-scale GLM-5 serving, a KV Cache race condition fix and LayerSplit optimization were disclosed.
A vLLM PR has added support code for Mistral-Medium 3.5 128B.