OpenAI Shares Initial Guidelines for Safety Cases in Frontier AI Training

OpenAI has shared initial guidelines for safety cases in frontier AI training, advocating for structured, evidence-based documentation before continuing any frontier reinforcement learning run. The company views these cases as an aspirational standard, aiming to bring the rigor of safety-critical industries like aviation and nuclear power to AI development, while acknowledging the unique challenges of emergent complexity in AI models.

Technical Safeguards

The proposed guidelines suggest safety cases should cover three aspects of the technical stack: alignment training, containment, and monitoring.

• Alignment Training: Focuses on preventing misaligned behavior through automated dataset review, manual checks, and grader tuning to penalize reward hacking. It also recommends offline alignment evals and worst-case stress tests to detect misalignment propensities.
• Containment: Involves hardening infrastructure security and conducting containment red-teaming using checkpoints from frontier runs. It recommends limiting high-bandwidth cross-sample communication and saving immutable transcripts for incident investigation.
• Monitoring: Recommends live systems with high recall on known issues and eval freshness to prevent stale metrics. It includes rapid response protocols, such as auto-pausing runs for unacknowledged alerts at night.

Operational Guidelines

Beyond technical controls, the guidelines outline operational best practices to ensure accountability and transparency.

• Dissents and Approvals: A member of another team should write a dissent to identify holes in the safety case. Senior leadership, including the Head of Safety and Chief Scientist, should review and have the ability to veto the run.
• Accountability and Pausing: The senior leader responsible for the run should be accountable for the safety case and incident response. Clear runbooks and SLAs should exist for pausing runs if new security issues invalidate the safety case.
• Technical Controls: Safety features like monitoring and auto-pausing should fail closed, meaning it should be impossible to start runs without appropriate monitoring enabled or to disable monitors from within training.

Incident Investigation

The guidelines also detail best practices for investigating severe AI misalignment incidents, emphasizing learning from failures to prevent recurrence.

• Root-Cause Analysis: Researchers should use targeted ablations or resampling experiments to understand how misaligned behaviors were introduced.
• Detection and Disclosure: New alignment testing methods should be developed to discover incident propensities without hillclimbing on incident data. Investigation results and postmortems should be shared with the public and affected third parties.