Why AI Alignment researchers want to automate themselves

The AI safety research workforce continues to grow, but it still accounts for a low share of overall AI research. Most resources remain focused on making models faster, smarter, and cheaper.

Frontier models from OpenAI, Anthropic, and Google DeepMind are already contributing to their own advancement, and are expected to continue improving themselves going forward. As AI's ability to train the next generation of models improves, there is a possibility that this could outpace the rate at which human researchers can control AI.

AI Alignment is the challenge of making AI act according to the user's intent. Because human oversight capacity has limits, automated Alignment research is essential to safely manage superintelligent AI.

OpenAI's Superalignment team set a goal of building a human-level automated Alignment researcher. Anthropic's Jan Leike is optimistic that it will be technically possible to create a model that performs Alignment research as well as a human while also being trustworthy.