Claude's Constitution

Existing model training methods relied on feedback where humans directly compared and selected between model responses. However, this approach has limitations: workers risk exposure to harmful content, and it becomes less scalable as models grow more complex.

Constitutional AI (CAI) was designed to solve these problems by having AI evaluate responses according to principles on its own, instead of relying on human feedback. The model follows an explicit set of principles called a Constitution, which helps it avoid harmful or discriminatory responses and build a system that is helpful, honest, and harmless.

The CAI training process largely proceeds in two stages:

• Stage 1: The model is trained to critique and revise its own responses based on given principles and examples.
• Stage 2: During the reinforcement learning process, AI-generated feedback is used instead of human feedback to train the model to select more harmless responses.

Through this approach, a Pareto improvement can be achieved, producing a model that is both more helpful and more harmless than existing methods. This is also regarded as a successful example of Scalable oversight, enabling model supervision without direct human intervention.