DiffusionGemma Technical Report

DiffusionGemma is an experimental open-weight language model that uses discrete diffusion to reduce the sequential token decoding bottleneck of traditional autoregressive (AR) LLMs. Instead of generating one token at a time, it generates text by iteratively refining blocks of 256 tokens in parallel.

The model was built by fine-tuning the Gemma 4 mixture-of-experts model rather than training from scratch. It has a total of 25.2B parameters, with 3.8B parameters activated during inference.

Training utilized a 2-stage compute-efficient pipeline based on existing AR models, with the report stating that it used less than 10% of the compute required to train the original AR model from scratch.