4-bit LLMs Surpass Original Model Performance

Applying structural compression and 4-bit quantization for the efficient deployment of large language models (LLMs) causes a degradation in inference and code generation capabilities. Existing healing techniques, such as Quantization-Aware Training (QAT), are costly and unstable, while Quantization-Aware Distillation (QAD) uses the bfloat16 checkpoint generated after structural compression as the teacher, limiting the performance ceiling.

The new Quantization-Aware Healing (QAH) technique directly distills from the original full-precision model before structural compression. Even if the teacher and student models have different architectures, information transfer is possible by matching output distributions via KL divergence, redefining the quantization step not merely as a loss step but as an additional distillation process from the original model.

Compressing the GPT-OSS 120B model to 60B parameters and quantizing it with MXFP4 resulted in higher performance than the original bfloat16 model on 7 out of 9 benchmarks. This approach demonstrates a reversal phenomenon where accuracy improves while reducing model size and cost, and it also proposes a memory-efficient chunk KL divergence loss function for long-context (32k token) processing.