AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
A router service has launched that reduces inference costs by running multiple models simultaneously and quickly selecting the optimal one.
Moonshot AI's Kimi K3 models are now supported through US-based providers on Vercel AI Gateway, including ZDR capability.
Baseten has launched an ultra-fast API dedicated to GLM-5.2 that maximizes performance by minimizing latency.
BeeLlama.cpp, a fork of llama.cpp, has released v0.4.1, which improves KV cache quantization efficiency.
Llama.cpp now supports MCP (Model Context Protocol), enabling the construction of agentic environments based on local models.
LoopGain, an open-source library based on control theory, automatically terminates repetitive AI agent loops to cut API costs by up to 92.8%.
Cloud infrastructure company Hetzner is preparing to build infrastructure for LLM inference.
Cactus added a 68k-parameter probe layer to Gemma 4 E2B and unveiled a hybrid inference system that uses the model's own error detection as a signal for cloud routing.
Claude Opus 4.8 has emerged as the dominant model, accounting for 40% of Anthropic's total token consumption and 45% of dollar spend on OpenRouter.
pi version 0.81.0 has added integrated support for llama.cpp (llama-server router).
Per-token prices have plummeted, but actual AI costs have risen instead due to a surge in usage driven by agent architectures.
Stoke, an open-source tool that supports efficient routing and management between Ollama instances running on multiple machines, has been released.
A study has found that quantizing Qwen2.5-Coder to 4-bit results in almost no degradation in code generation performance.
Research on ATSInfer, which greatly improves LLM inference performance on consumer devices through tensor-level scheduling, has been announced.
A new KV cache technique that stores verified knowledge as KV states and restores it without recomputation was announced.
ik_llama.cpp has added support for the openPangu-2.0-Flash model along with a GGUF format update.
We built an efficient scheduling pipeline that reallocates idle GPU resources, which occur due to traffic fluctuations in LLM services, for research and training purposes.
ExLlamaV3 v1.0.0 has been released, offering improved inference speed and support for a variety of models.
Qwythos-9B-v2, which applies FTPO technology to solve the repetitive generation problem, has been released.
A new technique is proposed that compresses narrative portions of the reasoning process while preserving only the computation/verification segments, reducing tokens while maintaining performance.
llama.cpp has fixed a bug where excessive context checkpoint creation during agent tasks degraded performance.
Mesh LLM has been released, unifying GPUs and memory distributed across multiple devices into a single OpenAI-compatible API.
It's an LLM server that maximizes inference efficiency on low-spec CPU environments through KV cache reuse and n-gram archives.
It maximizes LLM inference efficiency by dynamically adjusting the number of speculative decoding tokens according to the hardware's compute and memory bandwidth state.