DiffusionGemma Achieves 1,500 Tokens/s

Google DeepMind adapted the Gemma 4 26B A4B MoE checkpoint, originally trained autoregressively, into a discrete diffusion model without pre-training.

Instead of generating tokens one by one, DiffusionGemma denoises blocks of up to 256 tokens in parallel using bidirectional attention, refining them iteratively. This approach aims to reduce the memory bandwidth bottleneck of autoregressive models and shift inference into the compute-bound regime.

Training consists of two stages:

• SFT: Adapts the model to denoise 256-token noise blocks while referencing clean context.
• Sampler Distillation & Reinforcement Learning (SD·RL): Reduces forward passes and latency while maintaining generation quality.

Designed to reuse Gemma 4's multimodal understanding, long context, and thinking mode by leveraging the same transformer backbone. Reported evaluations show an average of approximately 20 tokens generated per forward pass, achieving ~1,500 TPS on H100 with FP8 and batch size 1.

This is 7.1x faster than the Gemma 4 AR model (204 TPS) and 4.8x faster than the AR model with MTP (303 TPS) in the same environment. It demonstrates that text diffusion models can leverage the knowledge and capabilities of existing AR checkpoints while reducing generation latency in low-concurrency environments.