Agentic GRPO: Optimizing Agent Training

Existing LLM reinforcement learning (RL) processes a single trajectory that runs from one prompt to a final answer. However, agent systems that repeat tool calls, code execution, debugging, and so on have very long trajectories, rewards arrive late, and training is very difficult due to the 'off-policy drift' problem where the policy changes during training.

Agentic GRPO is an algorithm that optimizes GRPO (Group Relative Policy Optimization) for agent workflows in order to solve these problems. GRPO groups multiple samples, ranks them relatively, and rewards the better trajectories.

The core innovation of this algorithm is the combination of immediate rewards and delayed correction.

• Immediate updates: The model is updated immediately whenever feedback occurs at an intermediate step.
• Retroactive correction: Once the final outcome is confirmed, previously performed updates are retroactively corrected.

This approach reduces the training instability that occurs when an agent performs long, multi-step tasks and enables efficient training.