TRL Supports Stable Diffusion RLHF with DDPO

Reinforcement learning (RL) is being introduced to address the alignment problem, where diffusion models like Stable Diffusion fail to fully align with human preferences or intentions.

DDPO (Denoising Diffusion Policy Optimization) frames the denoising process of diffusion models as a multi-step Markov Decision Process (MDP). Unlike the existing reward-weighted regression (RWR) approach, this has the advantage of optimizing complex objectives more efficiently and accurately by computing the exact likelihood of each denoising step.

Using the DDPOTrainer newly integrated into HuggingFace's trl library, users can leverage DDPO to fine-tune Stable Diffusion models to match human aesthetic standards or specific intentions.