Does a Transformer really need three projections? A systematic study of QKV variants

In the QKV (Query, Key, Value) attention structure at the core of Transformers, researchers analyzed the individual contribution of each projection and systematically studied the performance changes when they are shared.

The research team evaluated three projection-sharing constraints:

• Q-K=V (Key-Value sharing)
• Q=K-V (Query-Key sharing)
• Q=K=V (single projection)

Experimental results showed that the Q-K=V approach kept the perplexity degradation in language modeling to just 3.1% while achieving a 50% reduction in KV cache.

In particular, when combined with existing efficient attention techniques, this approach showed dramatic memory savings:

• Q-K=V + GQA-4: 87.5% reduction in KV cache
• Q-K=V + MQA: 96.9% reduction in KV cache

This efficiency is made possible by the fact that Key and Value occupy similar representation spaces, and attention computation takes place in a low-rank region. This research provides important evidence for designing efficient LLM inference models for on-device and edge devices.