LG AI Research 391

Recently, large language models (LLMs) have shown remarkable performance improvements through massive amounts of data, but supervised fine-tuning (SFT), which requires direct data from experts, has limitations in improving performance. To overcome this, research on RLHF (Reinforcement Learning from Human Feedback) and RLAIF (Reinforcement Learning from AI Feedback), which utilize human or AI feedback, is being actively conducted.

Existing RLHF goes through a standard two-stage process in which a reward model is first trained, and then a policy is trained based on it through an algorithm such as PPO (Proximal Policy Optimization). However, this method has the drawback that the training process is unstable and performance improvement is not easy.

Two major studies that recently drew attention at NeurIPS 2023 are as follows.

• DPO (Direct Preference Optimization): A methodology proposed by Stanford University that trains a policy directly from preference data without separately training a reward model. Through mathematical proof, it proposed a single optimization problem that can obtain an optimal policy equivalent to existing RLHF, and demonstrated more stable and higher performance across various benchmarks such as sentiment generation and summarization.

• Fine-Grained RLHF: A methodology proposed by the University of Washington that moves away from the existing approach of simply comparing preferences between answers, and instead clearly defines the metrics that need performance improvement while utilizing more fine-grained human feedback. This reduces noise arising from differences in preferences between individuals and provides more effective training signals for various metrics.