DiffusionGemma Transparency Audit (9-minute read)

GDM's (Google DeepMind) interpretability team conducted a transparency audit of a new text diffusion model, DiffusionGemma. The study found that DiffusionGemma performs similarly to the existing Gemma model on monitoring evaluations, and shows no significant difference in terms of variable transparency either.

Although DiffusionGemma's Opaque Serial Depth is theoretically much larger, the team confirmed that applying the Logit Lens to intermediate vectors and removing non-interpretive information does not cause any performance degradation. This means the intermediate nodes are interpretable, which brings the actual opaque serial depth down to the level of Gemma.

However, achieving Variable Transparency does not guarantee Algorithmic Transparency. Existing Autoregressive LLMs generate tokens sequentially, making the reasoning process easy to trace, whereas Diffusion models generate tokens simultaneously across the entire canvas, making it much harder to identify causal relationships between tokens.

The research team analyzed phenomena unique to text diffusion models, such as Non-chronological reasoning and Token smearing. In conclusion, while DiffusionGemma showed progress in understanding certain algorithmic styles, its algorithmic transparency is still lower than that of existing Autoregressive models.