PIRL: A Closed-Loop Optimization Technique to Prevent Performance Degradation in Reinforcement Learning

Most existing RL post-training algorithms such as PPO, GRPO, and DAPO are Open-Loop in nature, optimizing the local objective of the current batch and then immediately moving on to the next step. This approach carries the risk of the policy drifting in unintended directions or collapsing due to sampling noise or imperfect Credit Assignment.

The newly proposed PIRL (Policy Improvement Reinforcement Learning) implements Closed-Loop optimization by introducing a feedback signal that measures whether the updated policy has actually improved over the previous one. The key idea is to make actual Performance Gain between successive policies the direct optimization objective, rather than simply achieving a local objective.

PIPO (Policy Improvement Policy Optimization), a practical framework for implementing this, operates in two phases:

• Phase 1 (Exploration): Using an existing algorithm (PPO, GRPO, etc.), the policy is updated exploratorily according to the current local signal.
• Phase 2 (Retrospect): The actual performance change of the updated policy is checked, and the policy is reinforced or corrected based on this.

This approach has a Plug-and-play structure that can be immediately applied to various existing RL algorithms, enhancing training stability and strengthening alignment with final task performance.