Personalized Group Relative Policy Optimization for Heterogeneous Preference Alignment

Existing LLM alignment, like RLHF, optimizes a single global objective, so it fails to sufficiently reflect different individual preferences. In particular, the widely used GRPO assumes that samples within a group are interchangeable, which causes it to mix different users' reward distributions in personalized settings and bias training toward the majority preference.

To address this, we propose P-GRPO (Personalized GRPO). The core idea is to normalize the advantage not against the statistics of the current batch, but against the reward history of each preference group, reducing dependence on the immediately generated group.

This approach preserves distinct preference signals while retaining the contrastive learning signal, mitigating the phenomenon where sparse minority preferences disappear. In experiments across various tasks, P-GRPO converges faster and achieves higher reward than standard GRPO, showing that handling reward heterogeneity at the optimization level is key to personalized alignment.

• Problem: A single global objective struggles to properly reflect diverse user preferences.
• Cause: GRPO's group normalization mixes heterogeneous reward distributions into one, creating bias.
• Solution: P-GRPO, which normalizes the advantage based on the reward history of each preference group.
• Result: Faster convergence, higher reward, and better recovery of diverse preference signals.