SAPO: A Stable, High-Performance Reinforcement Learning Method for Training Large Language Models

SAPO(Soft Adaptive Policy Optimization) is a policy optimization method aimed at reducing the problem of large token-level importance ratio fluctuations in large language model reinforcement learning. It specifically targets situations where policy updates become noisy in Mixture-of-Experts(MoE) models or long autoregressive outputs.

Existing GRPO and GSPO use hard clipping, which cuts off gradients when the importance ratio exceeds a certain range. This approach prevents abrupt updates, but it causes problems such as discarding useful learning signals or ignoring an entire sequence because of just a few off-policy tokens.

SAPO replaces hard clipping with a temperature-controlled smooth gating function. Tokens whose policy has drifted far from the behavior policy are smoothly down-weighted, while still-useful gradients are preserved. This design maintains sequence-level coherence while providing token-level adaptivity.

The key differences are as follows:

• Reduces the discontinuity of clipping through a continuous trust region.
• Maintains sequence-level stability like GSPO, but does not easily discard the entire sequence.
• Selectively suppresses only problematic tokens through token-level adaptivity.
• Applies an asymmetric temperature design that uses different temperatures for positive/negative advantage.

The Qwen team explains that SAPO provides a better balance among stability, sample efficiency, and consistency of training progress. In conclusion, SAPO aims for more stable policy optimization in RL pipelines that strengthen LLM reasoning, coding, and multimodal reasoning.