KV cache down 35%

Running Qwen 3.6 long-context (1M) on an A100 40GB with HF Transformers, I inserted a small forward hook into the attention block to separately compress K and V.

• V is aggressively compressed with INT2/INT3 per-channel
• K keeps higher precision to avoid softmax instability
• The full KV cache is kept, with no eviction/token dropping

In the initial results, KV cache memory dropped from 10.7GB to 6.9GB, and looking at V alone, it shrank by about 3.5x.

Generation quality looked stable qualitatively, and in the early runs perplexity barely changed, but with only about 3 seeds it's still too early to draw conclusions.

The interesting points are as follows.

• V was far more compressible than expected
• At the same memory budget, rank reduction (SVD-style) degraded faster
• Since Qwen itself is already well-optimized, the savings were smaller than with other models

I'm still at the stage of finding out where it breaks, and I'd also like to see comparisons with approaches like kvpress, KIVI, H2O, as well as application cases on Llama 3 / Mistral.