AI BriefingKV Cache Quantization Performance Comparison
Key point
In Tesla P40 benchmarks, the saw-family KV quantization was the most stable.
Details
Teacher-forced PPL and token speed were compared based on ctx 512 on the WikiText-2 test set.
- The environment was Tesla P40 24GB, running
dist/bin/perplexityasturboquant-qjlfrom theollama-bugfixbranch. - Using the same methodology, PPL, KV MiB, total MiB, decode/prefill tok/s were measured for each preset.
On Llama 3.2 3B and Llama 3.1 8B, saw8k was effectively lossless. It was +0.0012 PPL for 3B and -0.0004 PPL for 8B, and saw8kv, q8kv, q8_0 were also nearly identical to f16. In contrast, 4-bit and TurboQuant-family presets showed rapidly growing loss.
On Qwen2.5 7B, attn_k.bias was large enough that the int4 step size overwhelmed the signal, and q4k, q4kv, saw4k, saw4kv, q4_0 all produced GARBAGE PPL. Only int8-family presets were in the normal range for this model.
On Qwen3.5 9B, most presets were near-lossless, but the K-only int4 encode path crashed due to a nil pointer bug. saw4kv worked fine since it uses a separate path, and q8kv and saw8k were also stable.
On Gemma4 27B, non-standard attention made teacher-forced PPL invalid as a quality metric, so effectively only whether VRAM was reduced mattered. f16 and some TurboQuant presets hit OOM, and only fused-path presets could run at all.
The overall conclusion is that the saw family, especially saw8k/saw8kv, is the safest and most reproducible choice.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
LLMTraceFX Introduces Independent Auditor to Verify That Cache Hits Do Not Prove Computation Omission
KV Cache Research: Limitations of LRU Alternatives and Failure in Capacity-bound EnvironmentsSalesforce Releases 'Random Attention' KV Cache Eviction Policy for Reasoning Models