AI BriefingAnthropic outlines best practices for robust AI safety evaluations
·2026.11.13 09:00
Key point
The research note recommends preregistration, domain expert involvement, and multi-turn simulations to address gaming, poor reproducibility, and saturation in current AI safety benchmarks.
Details
- Anthropic researchers propose a framework for building strong safety evaluations to guide deployment decisions, emphasizing that current AI evals are prone to gaming, misalignment, and lack of reproducibility.
- The core recommendation is to adopt lessons from psychology's replication crisis by implementing preregistration, rigorous statistical analysis, and improved measurement validity before data collection begins.
- Safety evaluations should use multi-turn simulations with domain experts to capture realistic escalation patterns, as single-turn tests often saturate (94-98% harmless response rate) and fail to detect subtle failures.
- Graders (LLM judges) must be validated against human labels using metrics like Cohen's kappa, and results should report confidence intervals to distinguish signal from noise, noting that API multi-turn eval confidence intervals can exceed ±10 percentage points.
-
Design and Execution
- Use observational designs for understanding behavior and causal designs for analyzing specific changes, ensuring construct validity (e.g., refusal is appropriate for fraud but a failure in suicide crises).
- Prevent data contamination by using data from after the model's training cutoff or synthetic data, and run cyber evaluations in sandboxed environments with real-time monitoring to prevent unintended actions.
-
Grading and Reporting
- Graders should use atomic, binary criteria and be tested for biases such as preference for longer responses or the first option in pairwise comparisons.
- Report results by category with confidence intervals; for example, the difference between Claude Opus 5 (98%) and Opus 5.5 (94%) in romance scam evals was not statistically significant.
-
Tools and Disclosure
- Utilize the UK AI Security Institute's Inspect framework and Anthropic's Petri toolkit for automating multi-turn behavioral tests.
- Disclose defects responsibly, such as the Claude Fable 5.1 suicide/self-harm eval issue where adding a system prompt instruction improved the appropriate response rate from 60% (API) to 94% (claude.ai).
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Author argues for urgent 6-12 month AI pause to mitigate 2028 superintelligence risks