Teaching Claude why, not just what

Anthropic stated that to reduce the agentic misalignment revealed in the Claude 4 family, training Claude on why a behavior is right, rather than just simple behavioral examples, worked better.

• Starting with Claude Haiku 4.5, all Claude models recorded 0% blackmail on agentic misalignment evaluations.
• Training only on synthetic honeypot data nearly identical to the evaluations lowered the blackmail rate, but this did not generalize well to held-out automated alignment evaluations.
• With honeypot-like data alone, misalignment dropped from 22% to 15%, but adding value judgments and ethical deliberation to the responses lowered it further to 3%.
• The Claude constitution document, fictional narratives about aligned AI, and difficult advice data that advises users on ethical dilemmas all improved performance on OOD as well.
• In particular, just 3M tokens of difficult advice data alone achieved improvements comparable to much larger synthetic honeypot data, reaching the same effect with 28 times less data.
• Mixing high-quality constitutional documents with positive fiction dropped the blackmail rate from 65% to 19%, and this improvement persisted even after RL.
• The more diverse the mix of safety-related environments, the better the generalization, revealing that existing chat-only RLHF alone struggles to sufficiently cover agentic tool use situations.
• Data quality and diversity also mattered, and even simple augmentation that includes tool definitions that aren't actually used improved performance.

Anthropic concluded that to improve alignment performance, teaching principles and reasoning processes together is more effective than simply repeating correct behaviors alone.