Is a Single Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Training Performance

RL (Reinforcement Learning) has become a core component of LLM post-training, but it has not been clearly established how RL adaptation is distributed across transformer layers. This study questions the conventional approach of uniformly updating all parameters, and demonstrates that training only a single transformer layer can recover most of the performance of full-parameter RL training.

The research findings show that performance improvements through RL are concentrated in specific layers.

• Layer Contribution: The researchers introduced a metric that measures how much of the full RL performance is recovered when only a specific layer is trained.
• Middle-Layer Concentration Phenomenon: Testing various models such as Qwen3 and Qwen2.5, and algorithms such as GRPO and GiGPO, they confirmed a pattern in which performance gains are concentrated in the middle portion of the transformer stack.
• Consistent Pattern: Layers close to the input and output ends showed lower contribution, and this structural pattern was found to be highly consistent regardless of dataset, task domain (math, code, agent, etc.), or model family.