Diffusers Adds LoRA Training Support for Stable Diffusion

LoRA (Low-Rank Adaptation) is a technique proposed to reduce the fine-tuning cost of large language models (LLMs); it freezes the pretrained weights and injects rank-decomposition matrices to greatly reduce the number of trainable parameters.

Applying this technique to the cross-attention layers of Stable Diffusion enables efficient learning of the relationship between image and text representations. LoRA training support has been officially integrated into Hugging Face's diffusers library, with the following key benefits:

• Faster training: Much faster than full model fine-tuning.
• Lower compute requirements: Efficient training possible even on a GPU with 11GB VRAM.
• Extremely small model size: Only a weight file of about ~3MB, roughly 1,000x smaller than the existing UNet model, needs to be saved.

This allows users to share only a lightweight LoRA file instead of the entire massive model when sharing fine-tuned models (e.g., Dreambooth), significantly reducing storage and download costs.