Hugging Face Unveils Quantization Backend 'Quanto'

Quantization is a technique that reduces a model's memory usage and computation cost by representing weights and activations in low-precision data types such as int8 or float8.

To solve the compatibility and device limitations of existing quantization libraries, Hugging Face has unveiled Quanto, a new PyTorch quantization backend for Optimum.

Key features of Quanto:

• Versatility: Supports Eager mode, so it works even with non-traceable models, and can be used on any device including CUDA and MPS.
• Automation: Automatically inserts quantization/dequantization stubs and quantization modules.
• Support for various precisions: Supports int2, int4, int8, float8 weights and int8, float8 activations.
• Performance optimization: Provides accelerated matrix multiplication (int8-int8, fp16-int4, bf16-int8, bf16-int4) on CUDA devices.
• Flexible workflow: Offers a seamless process leading from dynamic to static quantization, and is compatible with Safetensors.

After quantizing a model, users can compensate for performance degradation as needed through Calibration (recording activation ranges) or Quantization-Aware-Training (QAT).