AI BriefingSolving INT4 Collapse
Key point
The cause of KV cache INT4 collapse in some models was identified and fixed with a 4-line correction.
Details
An analysis was conducted on why INT4 quantization of the KV cache shows extreme performance differences across models.
- Falcon-40B: ΔPPL +0.08, almost no loss
- Qwen2-7B: ΔPPL +238, output essentially collapsed
- Pythia-6.9B: ΔPPL +22, Pythia-410M: ΔPPL +77, showing large model-to-model variation
The cause boils down to two factors.
- token-wise norm variation: In Pre-LN models, the KV vector norm fluctuates 2 to 5x across tokens, destabilizing row-wise absmax quantization
- activation outlier channels: Some channels are 8 to 100x larger than average, dominating the scale and ruining the precision of the remaining channels
Fixing only one side wasn't enough.
- Applying norm separation only: +57.5
- Applying per-channel only: +97.8
- Applying both together: improved to +0.32, about a 744x improvement
The core idea is to decompose the KV vector into norm (16-bit scalar) and direction (unit vector), then apply per-channel scaling to the direction. The proposed PyTorch code is about 4 lines, and it works as a drop-in preprocessing step placed in front of existing quantization, with no training, no calibration, and no per-model tuning.
In tests across 12 models (124M to 40B), the loss was greatly reduced in most cases, and models that already worked well were kept almost unchanged. Even the worst case, OPT-125m, only degraded to about ΔPPL +0.24.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.