Deliberative alignment: achieving safer language models through reasoning

Deliberative alignment, a new alignment strategy for OpenAI's o-series models, has been introduced. This approach is a paradigm in which the model is directly taught the text of human-written, interpretable safety specifications, and is trained to explicitly reason about those specifications before answering.

Existing methods such as RLHF (Reinforcement Learning from Human Feedback) or Constitutional AI used safety specifications only to generate training labels. Deliberative alignment, on the other hand, has the model use Chain-of-Thought (CoT) to review the user prompt, identify the relevant internal policies, and draft a safer answer.

This approach demonstrates the following key achievements:

• Precise policy compliance: It complies with OpenAI's safety policies with very high precision.
• Efficient training: Training is possible without human-labeled CoT or answer data.
• Overwhelming performance: The o1 model significantly outperforms existing state-of-the-art LLMs, including GPT-4o, on various safety benchmarks.

As a result, this serves as a case demonstrating that improvements in a model's reasoning ability can lead to improvements in safety.