NanoQuant: Efficient Sub-1-Bit Quantization for Large Language Models

NanoQuant is a post-training quantization (PTQ) method that compresses the weights of large language models (LLMs) to 1 bit or below. It compresses a 70B model on a single NVIDIA H100 GPU in 13 hours, reducing storage by 24x and enabling inference on 8GB consumer GPUs.

Existing 4-bit quantization is widely used in industry, but 1-bit quantization has a problem where actual storage cost balloons to 2-3 bits per weight due to metadata such as scale values and per-group parameters. On the other hand, 1-bit quantization-aware training (QAT) is expensive because it requires full training using billions of tokens and multiple GPUs.

NanoQuant addresses this problem by applying low-rank binary decomposition. It represents the weight matrix as two binary matrices and row- and column-wise scale vectors, reducing the amount of binary storage as well as metadata overhead. By choosing an appropriate low-rank dimension (rank), the total storage cost can be brought below 1 bit per weight.

The process of directly optimizing binary matrices is a combinatorial, non-convex discrete optimization problem. In particular, PTQ can only use small-scale calibration data at the level of 128 sequences and a limited optimization budget, making the quality of initialization critical.

NanoQuant's calibration process consists of three stages, including global Hessian-aware calibration and sequential block-wise reconstruction. The key is to secure a sufficiently good initialization so that competitive accuracy can be achieved with only lightweight calibration.