AI BriefingGemma 4 MTP Was Hidden, Then Community Dug It Up, Then Google Belatedly Added a Workaround
Key point
Google excluded Gemma 4's MTP feature from the public release, then began supporting it via an auxiliary model after the community discovered it.
Details
Google excluded the MTP (Multi-Token Prediction) architecture from Gemma 4 model's public weights (HuggingFace), then belatedly started supporting it through an auxiliary model after the fact was revealed via community reverse engineering.
While open-source developers were analyzing .litertlm files for edge devices, they discovered that an MTP structure was included that was absent from the standard model weights. MTP is a technology that predicts multiple tokens simultaneously in a single computation, and when combined with Speculative Decoding, it significantly improves inference speed without degrading output quality.
According to the community's analysis, applying MTP confirmed a speed improvement of about 3x (8 tps → 25 tps) compared to before in code generation tasks. This figure greatly exceeds the typical performance improvement range of ordinary speculative decoding (1.5x to 2x).
As community criticism intensified, in early May Google separately released a lightweight Drafter model of approximately 500M parameters called gemma4_assistant on HuggingFace. Through this, users can now utilize Gemma 4's MTP feature using the assistant_model argument without complex implementation.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Optimizing Gemma 4 Serving with vLLM on Amazon EKS [Part 2: Throughput and Latency SLOs on a Single GPU]
vLLM Publishes Performance Benchmarks for 5 Speculative Decoding Methods on AMD GPUs
3.85x Inference Speedup via Diffusion Draft Generation and AR Verification Based on GLM-OCR