AI Briefing#speculative-decoding
The latest AI and developer news about #speculative-decoding, with the original source and a short summary.
The latest AI and developer news about #speculative-decoding, with the original source and a short summary.
New CUDA mat-mul kernel for 2-24 src1 columns improves speculative decoding performance by 10.1% to 34.9% across tested NVIDIA GPUs.
The method saves 28 seconds on 120K-token prompts using ridge regression maps, with RULER scores dropping 1.2-1.4 points.
Removing unnecessary draft-token reservations allows larger prefill chunks and more efficient CUDA graph execution, raising output throughput from 1,373 to 3,390 tok/s.
The method achieves a 27.8% increase in mean acceptance length over DFlash on Qwen3-8B at temperature zero.
The method removes inter-round dependencies by drafting successor blocks during verification, achieving higher average speedups than DFlash across seven benchmarks.
Modular has released a free 57-page handbook covering LLM inference metrics, optimization, and operational costs.
The framework parallelizes draft sampling and supervision across independent branches to improve inference efficiency for large language models.
NVIDIA announced that Nemotron 3.5 Lightning generation speed on DGX Spark is up to 1.73 times faster when DSpark is applied.
An autonomous agentic loop discovers lossless system optimizations that nearly double the speedup of parallel drafters over autoregressive decoding.
A new method called Soft-MTP improves standard MTP by filtering out noisy gradients, resulting in lower perplexity and faster speculative decoding from 167M to 1.2B parameters.
The team combined mixed-precision quantization, DFlash2 speculative decoding, and custom CUDA kernels to reduce latency and improve throughput on a single GPU.
uzu, a Rust inference engine for Apple Silicon, recorded a speed of 142.3 tok/s on the M5 Max.
The method removes a noisy gradient component in standard MTP, improving perplexity and speculative decoding speed from 167M to 1.2B parameters.
The system uses high-fidelity production replay and parallel sub-agents to diagnose bottlenecks and test speculative decoding, quantization, and scheduling changes.
NVIDIA has released Nemotron 3.5 Bolt E4B, a small model for interactive agents that operates on 4GB VRAM.
The method outperforms DSpark by improving average throughput by 7.97% and acceptance length by 8.50% across seven benchmarks.
Combining block-level Flow generation with AR verification increased throughput by 2.54x.
The draft models combine a five-layer DFlash backbone with a learned bigram selector to propose multiple candidate continuations for target-model verification.
The system uses a lightweight quantized draft model to predict expert usage, reducing PCIe I/O bottlenecks on commodity hardware.
The open-source DeepInfer framework supports NVIDIA and Huawei Ascend platforms with a single-cluster throughput of 155 trillion tokens per day.
The framework reuses target model hidden states to avoid maintaining historical KV caches, outperforming DSpark by up to 70%.
The new DLoop method reduces target-model forward passes by performing multiple confident drafting stages before a single verification.
Speculative decoding accelerates LLM inference by increasing total FLOPs to utilize idle compute resources, shifting the workload from memory-bound to compute-bound at low batch sizes.
The training-free framework reaches 4.81x speedup on Huginn-3.5B by concurrently batching drafting and verification.