AI BriefingReQAT Achieves BF16 Accuracy with 4-bit Quantization for Weights, Activations, and KV Cache
·2026.11.05 07:49
Key point
The method identifies that FP4 errors stem from low-entropy tokens and uses trace-aligned training to reach 65.94% AIME-120 accuracy on R1-Qwen-14B.
Details
- ReQAT achieves 65.94% accuracy on AIME-120 for R1-Qwen-14B using NVFP4 W4A4KV4, matching the 65.46% of BF16 full fine-tuning while delivering 3.05x output-token throughput at 8K tokens.
- Analysis of over 1.5 million tokens reveals that FP4 quantization primarily causes errors at low-entropy positions (digits, operators) where the model is most confident, rather than at high-entropy branch points.
- The framework combines three components: Trace-Aligned QAT (TAQ) which retrains on the same traces to target fragile low-entropy predictions, Selective Entropy Minimization (SEM) which sharpens distributions for the lowest-entropy 75% of tokens, and Q-FIT which calibrates RoPE scaling and shifting per layer to minimize KV cache quantization error.
- Throughput gains for long outputs are driven largely by KV cache compression allowing larger batch sizes, as BF16 cannot sustain 256-request batches at 8K tokens due to memory limits, whereas NVFP4 can.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.