The Revival of Continuous Diffusion Language Models (CDLM) and Their Technical Background

Limitations of Autoregressive Models and the Emergence of Diffusion Models

Most modern language models use an autoregressive approach, generating tokens sequentially, one by one. While this method offers high training efficiency, it carries theoretical limitations such as exposure bias and difficulty in applying constrained generation tasks. To overcome these issues, there have been ongoing attempts to apply diffusion models, which have succeeded in image generation, to language.

Historical Flow of Continuous Diffusion Models

• 2021: Early studies adopted discrete diffusion methods to apply Gaussian noise to discrete data.
• 2022: Some studies, such as Diffusion-LM, introduced continuous diffusion methods by representing discrete categories as continuous embedding vectors and applying Gaussian noise. This had the advantage of allowing the direct use of tools designed for image generation.
• Late 2023: Subsequent research trends shifted back toward discrete diffusion, and research on continuous diffusion models effectively halted.

Recent Signs of Revival

After years of dormancy, new studies are emerging in the field of continuous diffusion models. This represents a new approach challenging the dominance of autoregressive models, gaining renewed attention particularly for its potential in controllable text generation.