Accelerating Gemma 4: Faster Inference with a Multi-Token Prediction Drafter

Weeks after releasing Gemma 4, Google announced that downloads have surpassed 60 million and unveiled an MTP drafter for the model family. This drafter is based on speculative decoding, proposing several future tokens first, which the target model then verifies in parallel to finalize the output.

The goal of this approach is up to 3x faster inference without quality loss. To reduce the problem where standard LLM inference is bound by memory bandwidth—moving huge weights through VRAM for every token—a lightweight drafter preemptively shares the computational load. When the target model accepts the draft tokens, it processes the sequence in a single forward pass, generating one additional token as well.

Google explained that the MTP drafter shares activations and the KV cache with Gemma 4 to reduce redundant computation. In particular, for the E2B/E4B edge models, efficient embedder clustering is applied to ease the bottleneck in final logit computation.

• Tested and optimized for: LiteRT-LM, MLX, Hugging Face Transformers, vLLM
• Official example: running Gemma 4 26B on an NVIDIA RTX PRO 6000 cuts latency roughly in half at the same quality
• Available on: Hugging Face, Kaggle
• License: Apache 2.0
• Usable via: transformers, MLX, vLLM, SGLang, Ollama, Google AI Edge Gallery

Google emphasized that this speedup can improve responsiveness for local development, agentic workflows, and on-device inference.