Dreambooth Training Optimization Guide Based on Diffusers

Dreambooth is useful for teaching new concepts or styles to Stable Diffusion, but it is prone to overfitting. HuggingFace presented a guide for obtaining optimal training results through experimentation.

Key Recommended Settings

• Learning Rate and Steps: It is important to use a low learning rate and gradually increase the number of training steps until satisfactory results are achieved. Finding the 'sweet spot' between learning rate and steps is key to avoiding overfitting.
• Face Training: When training faces, 800-1200 steps (based on batch size 2, LR 1e-6) is appropriate, and Prior Preservation techniques must be used to prevent overfitting.
• Addressing Quality Degradation: If generated images are excessively noisy, overfitting should be suspected. In this case, using the DDIM scheduler or increasing inference steps to around 100 is effective.
• Text Encoder Fine-tuning: Training the Text Encoder together with the UNet significantly improves quality. However, this requires more memory, so on a 16GB GPU environment, 8-bit Adam or fp16 training should be utilized.
• Token Selection: It is not necessary to use a rare token like sks; natural words that describe the target object can also be used without issue.