SDXL LoRA Training Optimization Script Released

Hugging Face has added a new training script to the diffusers library that combines the latest SOTA (State-of-the-Art) techniques to improve the quality of SDXL Dreambooth LoRA fine-tuning.

The core of this update is the combination of Pivotal Tuning technology used in Replicate's SDXL Cog trainer with the Prodigy optimizer from the Kohya trainer. This allows new concepts to be effectively learned even with a small number of images while maintaining SDXL's aesthetic quality.

Key Technical Features:

• Pivotal Tuning: Combines Textual Inversion, which inserts new tokens into the text encoder and optimizes them to avoid semantic interference with existing tokens, to increase training efficiency.
• Optimization Tools: Utilizes adaptive optimizers such as Prodigy.
• Other Recommendations: Supports various optimization parameters including text encoder learning rate adjustment, custom caption usage, dataset repeat count settings, and min SNR gamma application.

Users can use the new training script in diffusers or test it immediately via the provided Colab notebook. Additionally, for users who want to train through a UI without complex configuration, the 'LoRA Ease' service on Hugging Face Spaces is also introduced.