FLUX.1-dev, QLoRA Training on Low-Spec GPUs

This covers a QLoRA technique for efficiently fine-tuning the FLUX.1-dev model on consumer GPUs (under 10GB VRAM) using the diffusers library.

The key optimization techniques are as follows:

• Using QLoRA: Memory usage is dramatically reduced by loading the model quantized to 4-bit and then applying LoRA.
• Focused Transformer training: The text encoder and VAE are frozen, and only the Transformer component, the core of the model, is fine-tuned.
• FP8 optimization: FP8 training using torchao can further increase training speed on compatible hardware.

This guide provides results based on an NVIDIA RTX 4090 along with code that can be run immediately in Google Colab.