[NeurIPS 2023] RLHF, RLAIF Research Trends and Key Papers

Reinforcement learning methodologies that leverage human or AI feedback to improve LLM performance have recently been drawing attention. Existing RLHF (Reinforcement Learning from Human Feedback) goes through two separate stages—Reward Model training and Policy training—and this process has limitations in that training can become unstable or performance improvements can be difficult.

At this year's NeurIPS 2023, two key studies addressing these issues drew attention.

1. DPO (Direct Preference Optimization): A methodology proposed by Stanford University that learns the policy directly from preference data without training a separate reward model. It transforms the complex process of existing RLHF into a single optimization problem, showing higher performance and more stable training behavior.

2. Fine-Grained Human Feedback: A methodology proposed by the University of Washington that uses detailed feedback defined with clear metrics instead of simple preference information. This enables more effective training across various metrics, and it has demonstrated superior performance over existing methods on Detoxification and Long-Form QA tasks.

Going forward, research using such specific feedback signals to implement LLMs optimized for diverse user preferences is expected to accelerate further.