AI BriefingVirtual Head Attention Reduces KV Cache by 4x and Improves Quality Over GQA
·2026.10.24 00:59
Key point
VHA uses linear operations to halve physical query heads, achieving 4x KV cache reduction and +0.70% quality improvement over MLA on a 30B MoE model.
Details
- Proposes Virtual Head Attention (VHA) to resolve the trade-off between KV cache size, compute, and quality in long-context LLMs.
- VHA outperforms GQA-4 and MLA by +2.60% and +0.70% respectively on a 12-benchmark suite for a 30B-A3B MoE model.
- Achieves 4x KV cache reduction compared to GQA-8 at dense scales and 2x compared to GQA-4 at MoE scales.
- Reduces training and prefill FLOPs by 7.9% at the 1.7B parameter scale with 4096 context length.
- Uses Q Premix and Linear Postmix to recover diversity from halved query heads without nonlinear operations.
- Inference reduces to standard GQA-2 architecture, reusing existing GQA optimizations and exploiting Tensor Core parallelism with a group size of 64.
- Attains an average RULER score of 84.4 over 8K-128K context in a 10B-A1B MoE comparison, versus 73.0 for GQA.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
'Cache-to-Cache' Framework Revealed for Direct KV-cache Communication Between LLMs Without Text