AI BriefingThe KV Cache Skepticism
Key point
It argues that storing only the residual stream, without a KV cache, can produce identical outputs.
Details
Contrary to the common belief that a KV cache is essential for Transformer inference, it argues that keys and values can be deterministically recovered from each layer's residual stream.
The paper verified this across 6 models, 4 architecture families, and a parameter range of 135M to 4B, stating that the reconstruction error is at a bit-identical level.
It also observed D_KL = 0 in layer-wise cross-task residual patching, interpreting this to mean that the residual stream is effectively the sole information state and satisfies the Markov property.
Based on these results, it proposes KV-Direct.
- Checkpoints only the residual vector per token instead of the full KV
- Reduces memory to 5 KB/token on Gemma 3-4B
- Compared to the existing KV pair level of 136 KB/token
- In a 20-turn conversation, peak memory stays at 42 MB, while standard cache exceeds 103 MB
For comparison, it used H2O, StreamingLLM, SnapKV, TOVA, window-only as baselines, and states that KV-Direct maintained 100% token match across all cache budgets. It claims that, in contrast, the baselines' performance dropped to the 5–28% level.
It also analyzes that the recomputation approach is up to 5x faster than reading cached tensors for some operations, suggesting potential not only in memory savings but also in inference efficiency.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.