Improving Model Safety Behavior with Rule-Based Rewards

The existing RLHF (Reinforcement Learning from Human Feedback) method is central to improving a model's instruction-following ability, but it has limitations in that collecting human feedback for repetitive tasks is inefficient and it is difficult to respond when safety policies change.

To address this, OpenAI introduced Rule-Based Rewards (RBRs). RBR uses clear, step-by-step rules to evaluate whether a model's outputs comply with safety standards. This allows for balancing the prevention of harm while maintaining the model's usefulness, and enables efficient training without repeated human intervention.

RBR operates through the following process:

• Defining Propositions: Simple statements about specific aspects of a response, such as 'judgmental tone' or 'containing disallowed content,' are defined.
• Creating Rules: The defined propositions are combined to create sophisticated rules that can capture safe and appropriate responses across various scenarios.
• Utilizing a Grader: A fixed language model, the grader, scores responses according to the rules, allowing flexible adaptation to new rules or safety policies.

This technology has already been applied to GPT-4 and GPT-4o mini, and will continue to be implemented in future models to be released.