ProVer Framework Improves GRPO Credit Assignment for LLM Agents

Addressing GRPO's Uniform Advantage Flaw

Group Relative Policy Optimization (GRPO) is a promising method for training large language model agents, but it suffers from a credit assignment problem. By uniformly assigning trajectory-level advantages to all policy tokens, GRPO fails to distinguish consequential decisions from irrelevant ones, obscuring which intermediate steps contributed to success.

The ProVer Framework

To solve this, researchers introduced ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment. The process involves:

• Segment Proposal: An agentic judge contrasts successful and failed trajectories to identify segments likely responsible for divergent outcomes.
• Verification: Instead of trusting the judge directly, ProVer verifies the segment's importance by estimating its causal impact through counterfactual rollouts using the current policy.
• Advantage Update: The verified causal credit is integrated into the GRPO advantage calculation, allowing the model to focus learning on critical decision points.

Performance and Efficiency

Experiments on ALFWorld and WebShop benchmarks demonstrate that ProVer significantly outperforms standard GRPO and other baselines. Specifically, it achieves a 9.91% relative improvement in success rate for Qwen3.5-2B and a 7.12% improvement for Qwen3.5-4B. The framework also shows robustness, maintaining performance gains even when using smaller judge models, indicating that the verification mechanism effectively filters out noisy or incorrect segment proposals.