Recent Trends in LLM Architecture: KV Sharing, mHC, Compressed Attention

Recently released open-weight LLMs are focusing on long-context efficiency. As agentic and reasoning-style workflows hold onto tokens for longer, KV-cache size, memory traffic, and attention cost become the bottleneck.

Setting aside dataset composition, training schedules, post-training, RL recipes, and benchmarks, this piece looks only at the design changes made within the transformer block, residual stream, KV-cache, and attention computation. On the surface these look like small diagram-level differences, but the actual goal is structural changes that reduce the cost of processing long contexts.

Key cases are as follows.

• Gemma 4 E2B/E4B introduces a shared KV cache and per-layer embeddings, allowing later layers to reuse the key-value state of earlier layers. This direction connects with cross-layer attention research, and can be seen as an extension of the existing trend where GQA shares KV heads.
• The Gemma 4 lineup is split into E2B/E4B for mobile/embedded use, a 26B MoE for efficient local inference, and a 31B dense model aimed at quality and post-training convenience.
• Laguna XS.2 uses layer-wise attention budgeting to allocate attention resources differently across layers.
• ZAYA1-8B adopts compressed convolutional attention.
• DeepSeek V4 combines mHC with compressed attention.

Ultimately, the common direction in recent LLM architecture is shifting away from simply building bigger models, toward designs that handle longer contexts with less memory.