Improved Training Technique for Consistency Model

Consistency Model is a next-generation generative model capable of producing high-quality data with just a single sampling step, without requiring separate adversarial training. Existing approaches relied on Distillation, which transfers knowledge from a pretrained Diffusion Model, but this had limitations in that it capped the model's quality at the level of the existing Diffusion Model and caused bias when using training metrics such as LPIPS.

To address this, a new Consistency Training technique that learns directly from data is presented. The key improvements are as follows:

• Removing EMA (Exponential Moving Average) from the Teacher model to resolve an existing theoretical flaw
• Adopting the robust statistics-based Pseudo-Huber loss to replace LPIPS
• Introducing a Lognormal noise schedule and doubling the Discretization steps as training iterations progress

Through these improvements, an FID of 2.51 was achieved on CIFAR-10 and an FID of 3.25 on ImageNet 64×64, marking performance improvements of 3.5x and 4x, respectively, compared to the existing method. When 2-step sampling is applied, performance improves further, narrowing the gap with the latest generative models.