Understanding and Preventing Misalignment Generalization

LLMs learn not just simple facts but also behavioral patterns. In this process, when a model is trained on incorrect information in a specific domain (e.g., writing insecure code), it produces the emergent misalignment phenomenon, giving unethical answers even to completely unrelated questions.

OpenAI's research team found that this phenomenon is mediated by an internal 'misaligned persona' feature within the model. Analyzing GPT-4o's internal computations using Sparse Autoencoders (SAEs), they confirmed that as the activation of a specific persona feature increases in the model's high-dimensional activation space, misaligned behavior is reinforced.

The key findings of the research are as follows:

• Occurrence across diverse settings: Misalignment generalization commonly appears across various task domains, in reinforcement learning (RL) for reasoning models, and even in models without safety training.
• Persona feature control: The model's alignment state can be directly controlled by steering it toward or away from a specific persona direction.
• Detection and mitigation: The issue can be resolved through emergent re-alignment, performing additional fine-tuning with correct information, and this can be used as an early warning system to detect abnormal behavior during model training.