AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
Liquid AI has released LFM2.5-VL-3B-DSpark, a draft model for speculative decoding of its VLM LFM2.5-VL-3B, which improves decoding speed by up to 3.13x on the Apple M5 Max.
A community-developed Jinja template for GPT-OSS fixes a bug in the widely used Unsloth version that caused the model to drop final answers when reasoning traces were present in chat history.
Hayder Tirmazi's optimizations for prompt lookup decoding in llama.cpp, combined with a PR by Daniel Lemire, achieve up to 140x faster drafting and up to 2.6x less memory usage.
The new architecture maintains within 3.4% of native token throughput on NVIDIA H100 GPUs while enabling support for out-of-tree accelerators and older hardware.
Released the open-source tool entail to verify and prevent the issue where vLLM's rope_scaling override bug causes GSM8K scores to drop by up to 28%.
The new TQ method achieves a mean KLD of 0.0282 and 92.4% top-1 accuracy on Qwen 3.8 27B at 4-bit without requiring calibration data.
Underdog has released 'Husky', a model-specific inference engine that is faster than MLX across all measured metrics, achieving up to 4.5x performance gains particularly in editing tasks.
Inco AI has released Splash, an Apple Silicon inference engine that applies model-specific kernels and draft models.
This analysis compares the features and suitable workloads of major open-source inference stacks such as llama.cpp, vLLM, and SGLang.
GraphSignal, an open-source profiler that allows AI agents to directly read GPU performance data and execute optimization loops, has been released.
Intel has released OpenVINO 2026.4, adding support for the latest models such as Gemma 4 and Qwen 3.5/3.6, along with Multi-Token Prediction inference acceleration.
A single-file C-based inference engine has been released that performs text and image inference for Gemma 1/2/3 without external dependencies.
Edge0 released an inference framework that runs 35B MoE models on 3GiB of memory using SSD expert streaming technology.
At Hot Chips 2026, Micron analyzed the gap between compute and memory bandwidth and revealed that Samsung's LPDDR5X PIM technology improved Llama 3.1 8B inference speed by 3.01x.
LLMTraceFX has introduced an independent auditor to verify the reliability of cache hits, establishing evidentiary standards for whether computation is omitted.
NVIDIA has released 'PAIR', an open-source tool that connects multiple local computers to distribute AI inference requests.
RTK's token savings metrics diverge from actual cost measurements, with the latest models showing negligible or even increased costs.
Nvidia has released SoL-Pi, an extension featuring four mechanisms to reduce token usage and latency for Pi agents.
Research results have been published demonstrating that no KV cache policy outperforms LRU in simulations of real agent workloads.
Yandex researchers unveiled a new inference technique that modifies the KV cache to enhance LLM interactivity.
Salesforce has released Random Attention, which applies random sampling instead of signal-based methods for KV cache eviction in reasoning models.
The NInfer fork supports a 555k token context window on the RTX 5090 via NVFP4 KV cache and YARN extension.
The Rust-based CPU-only LLM inference framework eLLM has been released in beta, claiming to offer faster performance than GPUs for long inference tasks.
Cerebras offers GPT OSS and Qwen 3.8 models on public endpoints, specifying that all models are original, non-pruned versions.