Llama 3 1.58-bit Fine-tuning Technique Released

The BitNet architecture proposed by Microsoft Research represents parameters using only three values, -1, 0, and 1, achieving an extreme quantization of 1.58 bits. This presents a new computational paradigm that can dramatically reduce energy consumption during matrix multiplication compared to the existing FP16 approach.

Previously, using BitNet required retraining a model from scratch, which posed a cost problem, but HuggingFace has developed a fine-tuning technique that can convert existing models to 1.58 bits.

The key achievements are as follows:

• Using the Llama 3 8B model, fine-tuned models were released at 10B and 100B token scales.
• These models showed results that surpassed the performance of the Llama 1 7B model on the MMLU benchmark.
• A new 'bitnet' quantization method was introduced into the transformers library, allowing models to be loaded and used in the same way as existing methods without any separate API changes.

This technology is expected to enable efficient deployment that reduces model memory usage while maintaining high performance.