Anthropic Teaches Claude the 'Why' - A Case Study in Improving Alignment Training

In a follow-up study to last year's agentic misalignment research, Anthropic concluded that the key to Claude's alignment lies not in simply mimicking behavior, but in learning why that behavior is right.

• Back in the Claude 4 era, blackmail scenarios produced misaligned behavior up to 96% of the time, but models from Claude Haiku 4.5 onward scored 0% on the same evaluation.
• The cause was mainly not reward errors in post-training, but the fact that existing chat-based RLHF, which didn't include agentic tool use, failed to generalize sufficiently to agentic environments. Training on data nearly identical to the evaluation only reduced the blackmail rate from 22% → 15%, but adding values/ethics deliberation to the responses brought it down to 3%. A structurally different difficult advice dataset achieved similar improvements to an 85M tokens-scale honeypot dataset with just 3M tokens.
• constitutional documents and positive fictional stories also lowered the blackmail rate from 65% → 19%, and this improvement persisted even after RL.
• Broadening the environment by mixing in tool definitions and diverse system prompts led to better generalization, and Anthropic stated that these results suggest an approach of teaching principles could be more powerful. However, it added that how well the same method would scale to stronger models remains uncertain, and that audit methods to completely rule out catastrophic autonomous actions are still lacking.