Accelerating Gemma 4: Faster Inference with Multi-Token Prediction Drafters

Google released Multi-Token Prediction (MTP) drafters for the Gemma 4 family. It stated that specialized speculative decoding delivers up to 3x inference speed while preserving output quality and reasoning logic.

Since standard LLM inference is bottlenecked by memory-bandwidth constraints, a lightweight drafter first proposes multiple tokens, and the heavy target model verifies them in parallel. If correct, the entire draft is accepted and additional tokens are generated, producing longer output in a single forward pass.

• Improved responsiveness: Reduces latency for real-time chat, voice, and agentic workflows.
• Local/edge use: The 26B MoE, 31B Dense, E2B, and E4B models can run faster on PCs, consumer GPUs, and on-device environments.
• Architecture optimization: Leverages the target model's activations and shares the KV cache to reduce redundant computation.
• Additional improvements: Embedder clustering was applied to E2B/E4B to ease the final logit computation bottleneck.
• Hardware observations: It explained that the 26B MoE achieved about 2.2x faster local speed on Apple Silicon at batch size 4-8, and confirmed similar gains on Nvidia A100.

The models are provided under the Apache 2.0 license and are immediately available on Hugging Face, Kaggle, transformers, MLX, vLLM, SGLang, Ollama, and Google AI Edge Gallery (Android/iOS).