Google Clarifies the Principle Behind Diffusion Models' Creativity

A recent paper from Google Research (ICLR 2026) mathematically analyzes how diffusion models generate new samples, going beyond simply 'memorizing' and replicating training data.

The key lies in the properties of the Score Function. If a model perfectly learns the Empirical Score Function (ESF), which is the exact distribution of the training data, the model becomes nothing more than a retrieval tool that simply copies the data. In contrast, Regularization, which occurs during neural network training, plays the role of smoothing the score function.

This 'smooth score' produces the following effects:

• Interpolation Effect: It connects sharp data points like a gentle slope, enabling the generation of new values between training data points.
• Source of Creativity: Even without artificial smoothing, regularization such as weight decay in neural networks naturally smooths the score, guiding the model to interpolate between data points and create new samples.

In conclusion, the study demonstrates that the creativity of diffusion models stems from the process of interpolation between data through a smooth score function created by the model's 'imperfect learning.'