Mix-Quant: Quantize the Prefill, Keep the Decoding at High Precision

A study has been published that focuses on the fact that the two core stages of LLM inference, Prefilling and Decoding, have different computational bottlenecks and quantization characteristics.

The prefill stage processes the input sequence in parallel, making it suitable for applying aggressive quantization such as W4A4 (4-bit weight/activation). Quantization errors do not recursively affect subsequent inputs, and by leveraging redundancy within long contexts, a theoretical speedup of about 4x can be expected.

On the other hand, the decoding stage is highly sensitive to errors due to its autoregressive nature, where each token affects the next generation. Applying low-precision quantization to decoding carries a high risk of accumulated errors degrading generation quality.

To address this, the proposed Mix-Quant presents a hybrid approach that uses low-precision quantization during context encoding (prefill) to increase computation speed, while maintaining a high-precision path during token generation (decoding) to preserve accuracy.