AI Briefing#speculative-decoding
The latest AI and developer news about #speculative-decoding, with the original source and a short summary.
The latest AI and developer news about #speculative-decoding, with the original source and a short summary.
BeeLlama.cpp, a new fork that maximizes the performance of llama.cpp, has been released.
Google released an MTP drafter for Gemma 4, claiming up to 3x faster inference.
MTP support in llama.cpp can boost inference speed for the Qwen 3.6 27B model by 2.5x.
Google excluded Gemma 4's MTP feature from the public release, then began supporting it via an auxiliary model after the community discovered it.
A UCSD research team achieved an average 3.13x speedup with DFlash on Google TPU v5p.
z-lab has released the DFlash version of Gemma 4 31B-it on Hugging Face.
DFlash for Qwen3.5-35B-A3B was confirmed running on an RTX 2080 SUPER 8GB.
Applying Speculative Decoding to NeMo-RL boosted RL rollouts by 1.8x.
Luce DFlash boosted Qwen3.6-27B throughput by up to 2x on a single RTX 3090.
Applying ngram-mod to Qwen3.6 27B on the R9700 showed speed improvements, though with some variability.
With llama.cpp ngram-mod speculative decoding, Qwen-3.6-27B speed rose to 136.75 t/s.
TurboQuant fixed a speculative decoding bug in its Qwen3.6 fork.
MoE models achieve higher inference acceleration when Speculative Decoding techniques are applied.
FlashDrive optimizes the entire autonomous driving VLA inference pipeline, cutting latency from 716ms to 159ms.
Speculative checkpointing has been merged into llama.cpp, and speed improvements have been confirmed in some tasks.
On DGX Spark, Qwen3.6 performs similarly to Qwen3.5, but MTP significantly increases throughput.
SwiftLM presented results of running a 35B Qwen model on a 24GB Mac with 40K~100K context.
Cloudflare accelerated inference for massive LLMs through PD disaggregation, prompt caching, and KV-cache optimization.
Aurora is an open-source RL-based framework that continuously improves an LLM's speculative decoding performance by learning from real-time inference data.
OpenVINO and depth-pruning boosted Qwen3-8B inference speed by 1.4x on Intel Core Ultra.
A technology has been developed that can speed up LLM inference using any small model, even if the tokenizer is different.
Dynamic speculative decoding, developed by Intel Labs and Hugging Face, has been applied to the Transformers library, boosting generation speed by up to 2.7x.
It introduces how to implement a custom handler that combines Whisper and Pyannote in Hugging Face Inference Endpoints to support speaker diarization and speculative decoding.