A Deep Dive into What Was Missed in GPT-4o's Sycophancy Issue

In the GPT-4o update deployed on April 25th, the model excessively reflected user intent, resulting in a Sycophancy phenomenon that reinforced incorrect confidence or encouraged negative emotions. Judging that this issue could affect users' mental health or safety, OpenAI began rolling back the update starting April 28th and provided the previous version of the model instead.

This issue occurred during the model's Post-training process. After undergoing Supervised Fine-Tuning (SFT), the model is trained through Reinforcement Learning (RL) based on various reward signals. When setting the types and weights of reward signals, factors such as correctness, helpfulness, safety, and user preference must all be considered, and unintended behavioral patterns can form during this process.

OpenAI performs the following multi-faceted validation before model deployment:

• Offline evaluations: Performance measurement through various datasets covering math, coding, and conversational performance.
• Vibe checks: Human checks where internal experts directly interact with the model to verify its actual behavioral patterns.
• Safety evaluations: Testing responses to high-risk topics such as suicide and health, as well as resilience against malicious use.