Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Existing reinforcement learning (RL) pipelines for LLMs mainly operate in a Synchronous and Batch-interleaved manner, making them inefficient for Agentic tasks that involve long-horizon operations. Asynchronous RL, which has recently emerged as an alternative, offers faster model update speed but has limitations in training stability and task effectiveness.

This paper proposes SAO (Single-rollout Asynchronous Optimization) to address these issues. Noting that the group-wise sampling approach of the existing GRPO framework is not well suited for asynchronous training, SAO introduces Single-rollout sampling, which uses only one rollout per prompt. This reduces off-policy effects and improves generalization performance.

To secure optimization stability, SAO also includes the following technical devices:

• Introduction of a Strict double-side token-level clipping strategy
• Application of a practical Value-model training design

Experimental results show that SAO consistently outperformed GRPO and its variants on agentic coding and reasoning benchmarks such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. In particular, it showed excellent performance even in online learning environments that require adapting to changing conditions, and it was successfully deployed in the actual agentic RL pipeline of the GLM-5.2 (750B-A40B) model.