Qwen Team Unveils Head-Level Hybrid Attention 'HydraHead'

Unlike existing layer-level hybrid attention approaches, HydraHead proposes a new architecture that mixes Full Attention (FA) and Linear Attention (LA) at the attention head level.

The key innovations are as follows:

• Interpretability-based head selection: Identifies heads that play an important role in information retrieval and applies a strategy that keeps only those heads as FA.
• Scale-normalized fusion module: Adjusts the distributional differences between FA and LA head outputs to effectively combine the two signals.
• Efficient training pipeline: Achieves a high-performance model with minimal training overhead through a 3-stage transfer pipeline that includes parameter reuse and distillation.

Experimental results show that HydraHead, after training on only 15B tokens, achieved a performance improvement of over 69% compared to the baseline at a 512K context length, a figure close to the performance of similarly-sized Qwen3.5 models.