GSPO for Scalable Reinforcement Learning of Language Models

GSPO (Group Sequence Policy Optimization) is a new algorithm for scaling reinforcement learning (RL) of language models to be larger and more stable. Existing GRPO becomes unstable during long-duration training and can even cause irreversible model collapse, leading to a problem where performance does not keep improving even with more compute.

The core of GSPO is defining the importance ratio based on sequence likelihood rather than token-level, and performing sequence-level clipping, rewarding, and optimization based on that criterion. It also applies length normalization to reduce variance and stabilize the numerical range.

The authors explain that GSPO outperforms GRPO in the following three aspects.

• Performant and Efficient: It achieves higher performance at the same training cost, and performance keeps improving as compute is increased.
• Notably Stable: It remains stable even during long-duration training, particularly resolving RL training stability issues in MoE models.
• Infrastructure-Friendly: Thanks to sequence-level optimization, it is more tolerant of precision differences, which is likely to simplify infrastructure.

Experiments were conducted by fine-tuning a cold-start model based on Qwen3-30B-A3B-Base, comparing performance curves and training reward curves on AIME'24, LiveCodeBench, CodeForces against GRPO. As a result, GSPO showed higher training efficiency, demonstrating genuine scalability where performance kept improving as compute was increased, the query set was updated, and generation length was increased.

Notably, in GSPO, the clipped token ratio was more than an order of magnitude higher than in GRPO, yet it still achieved better training efficiency. This supports the point that while the token-level objective is noisy and inefficient, the sequence-level approach provides a more reliable training signal.

The change is also significant on the MoE RL front. In GRPO, training failed to converge due to expert activation volatility, requiring Routing Replay, but this approach incurred additional memory and communication costs. Since GSPO only looks at sequence-level likelihood, Routing Replay can be completely eliminated, resulting in simpler and more stable training while more fully utilizing MoE capacity.

GSPO is also less sensitive to precision discrepancy than token-level likelihood, suggesting the possibility of directly using likelihoods returned by the inference engine for optimization. This is particularly advantageous in environments such as partial rollout, multi-turn RL, and training-inference disaggregated frameworks.