AI Briefing#speculative-decoding
The latest AI and developer news about #speculative-decoding, with the original source and a short summary.
The latest AI and developer news about #speculative-decoding, with the original source and a short summary.
The release includes DFlash draft models for the 9B, 35B-A3B, and 397B variants of Ornith-1.5.
Liquid AI has released LFM2.5-VL-3B-DSpark, a draft model for speculative decoding of its VLM LFM2.5-VL-3B, which improves decoding speed by up to 3.13x on the Apple M5 Max.
Hayder Tirmazi's optimizations for prompt lookup decoding in llama.cpp, combined with a PR by Daniel Lemire, achieve up to 140x faster drafting and up to 2.6x less memory usage.
The new 280M-parameter draft model accelerates LFM2.5-VL-3B inference by up to 3.13x on device and 2.66x on H100 GPUs.
This article introduces vLLM optimization strategies to maximize Gemma 4's SLO-compliant throughput through NVFP4 precision and the separation of interactive and batch pools.
vLLM released detailed benchmark results comparing the throughput and optimal settings of five speculative decoding methods, including Native MTP, EAGLE-3, and DFlash, on AMD MI300X/MI355X GPUs.
A technique combining Diffusion draft generation and AR verification with the GLM-OCR model has been released, increasing decoding speed by 3.85x.
Introduces techniques for managing the latency and throughput tradeoffs in LLM inference and expanding the efficient frontier through quantization, speculative decoding, and more.
vLLM v0.28.0 adds large-scale performance optimizations and inference capabilities for the Kimi-K3 and DeepSeek V4 models.
Applying DSpark speculative decoding to LFM2.5 models improves GPU and on-device inference speed by up to 3.2x.
Inco AI released DFlash 2, a parallel inference acceleration technology that improves throughput by 16–25%.
ARBITRAGE predicts model superiority to reduce mathematical reasoning latency by up to 2x.
DiffusionGemma adapts Gemma 4 into a discrete diffusion model, achieving 1,500 TPS on H100.
FuriosaAI presented four papers at ICML 2026 addressing efficiency in LLM inference, training, and serving.
The VIDRAFT team boosted Gemma inference speed to 510.58 TPS on an A10G while maintaining quality.
OpenAI presented a strategy to lower AI costs and boost efficiency from infrastructure to products.
llama.cpp has been changed to load MTP tensors for certain architectures by default even when MTP is disabled, increasing VRAM usage.
The open-source llama.cpp project has added a DSpark-based Speculative Decoding feature.
Baseten has launched an ultra-fast API dedicated to GLM-5.2 that maximizes performance by minimizing latency.
This piece examines the latest techniques for optimizing inference efficiency and resource usage in LLMs, centered on research presented at ACL 2024.
MTP-based speculative decoding and support for Hy3, Tencent's 299B MoE model, have been added to llama.cpp.
It's an LLM server that maximizes inference efficiency on low-spec CPU environments through KV cache reuse and n-gram archives.
It maximizes LLM inference efficiency by dynamically adjusting the number of speculative decoding tokens according to the hardware's compute and memory bandwidth state.
A technique is proposed that maximizes the efficiency of Diffusion-based Speculative Decoding by predicting the optimal block size according to the input data.