4-bit Quantization and QLoRA Released

Hugging Face, in collaboration with the bitsandbytes library, has released a feature that enables models to run in 4-bit precision. This can be applied to Hugging Face models across various modalities, including text and vision.

In particular, the newly introduced QLoRA (Quantized Low-Rank Adaptation) method maximizes memory efficiency to the point where a 65B-parameter large model can be fine-tuned on a single 48GB GPU. QLoRA maintains performance on par with 16-bit fine-tuning while using the following three key technologies:

• 4-bit NormalFloat (NF4): A new data type that is information-theoretically optimal for normally distributed weights.
• Double Quantization: Further quantizes the quantization constants themselves to reduce the average memory footprint.
• Paged Optimizers: A technique for managing memory spikes.

With this technology, users can fine-tune or perform inference on large models such as GPT-neo-X (20B) even in constrained environments like Google Colab.