Recent Trends in LLM Architecture: KV Sharing, mHC, and Compressed Attention

Major open-weight LLMs released between April and May 2026 have focused on efficiency in long-context processing, introducing new architectural techniques to reduce KV cache size, memory traffic, and attention costs.

Gemma 4 introduced cross-layer KV sharing (cross-layer attention), allowing later layers to reuse KV tensors from earlier layers. E2B computes its own KV for only 15 of 35 layers and reuses the rest for the others, reducing the KV cache by roughly half. At a 128K context, E2B saved 2.7 GB and E4B saved 6 GB. Per-layer embeddings (PLE) kept the main transformer computation small while adding capacity through lookup-based embedding tables.

Laguna XS.2 (Poolside) applied layer-wise attention budgeting, allocating a different number of query heads per layer. Sliding window layers use 8 query heads per KV head, while full attention layers use 6, assigning a smaller budget to the more expensive global attention.

ZAYA1-8B (Zyphra, trained on AMD GPUs) uses Compressed Convolutional Attention (CCA), which compresses Q, K, and V all together and performs attention directly in the compressed latent space. Unlike MLA, it reduces not only the KV cache but also attention FLOPs during prefill and training, applying convolutional mixing to the compressed Q/K to compensate for local context.

DeepSeek V4 introduced two key innovations:

• mHC (Manifold-Constrained Hyper-Connections): expands a single residual stream into multiple parallel streams, while constraining the mapping to a manifold of doubly stochastic matrices to ensure scaling stability. In 7B experiments, it reached baseline performance with half the training tokens while keeping FLOPs nearly the same

• CSA/HCA compressed attention: compresses along the sequence dimension to summarize multiple tokens into fewer KV entries. CSA applies sparse selection at 4:1 compression, while HCA applies dense attention at 128:1 compression, complementing each other. At a 1M token context, compared to V3.2, V4-Pro reduced FLOPs by 27% and KV cache by 10%, while V4-Flash reduced FLOPs by 10% and KV cache by 7%

All of these architectures focus on lowering long-context inference costs without reducing total parameter counts. While the basic transformer skeleton remains unchanged, complexity has increased significantly—implementations that used to take 50-100 lines of PyTorch now take roughly 10x more.