DiffusionGemma Technical Report

DiffusionGemma, released by Google researchers, is an experimental open-weight language model that maximizes text generation speed by applying discrete diffusion.

To address the bottleneck of existing autoregressive (AR) models generating tokens sequentially one by one, this model adopts a method of iteratively refining 256-token blocks in parallel. It was built by fine-tuning the Gemma 4 MoE model (3.8B active, 25.2B total parameters), implementing an efficient two-stage training pipeline using less than 10% of the total training token budget of the initial AR model.

Performance and Efficiency

• Generates approximately 1,500 output tokens per second on a single NVIDIA H100 GPU.
• Generates an average of 20 tokens per forward pass, significantly faster than AR models equipped with the latest speculative decoding.
• Sets a new Pareto frontier in the trade-off between generation speed and model performance.

Other Features

• Retains core features of the original model, such as 'thinking mode', multimodal input, and long context support.
• Remains capable of AR generation despite diffusion fine-tuning, with negligible performance degradation, suggesting the possibility of transitioning to hybrid diffusion-AR decoding.