Researchers Warn AI R&D Automation Could Trigger Intelligence Explosion

The Premise: Automating AI R&D

A new research paper assesses the possibility of an intelligence explosion triggered by the automation of AI research and development (R&D). The authors note that AI systems currently write most of the code inside companies building them. As more of the R&D pipeline becomes automated, the paper suggests that AI progress could radically accelerate, compressing years of advances into months or less.

Potential Impacts and Risks

The paper outlines both the potential benefits and severe risks of this acceleration:

• Accelerated Benefits: An intelligence explosion could dramatically bring forward the benefits of AI.
• Loss of Control: Capabilities growth could accelerate far beyond society's ability to keep up, potentially leading to humanity losing control over superhuman AI systems.
• Erosion of Power Checks: Checks on power within and between states, companies, and branches of government could be severely eroded.

Policy Recommendations

Despite significant uncertainty, the authors argue the high stakes warrant serious attention. They propose that policymakers should:

• Urgently obtain more visibility into the automation of AI R&D.
• Develop methods to steer and constrain an intelligence explosion.
• Prepare society to adapt to the impacts of such an event.

Authorship

The paper is co-authored by prominent AI figures including Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, Jack Clark, Hilary Greaves, Anton Korinek, Samuel Hammond, Thore Graepel, Ben Bariach, Philip H. S. Torr, Sheila A. McIlraith, Jeff Clune, Sam Manning, Girish Sastry, Tom Davidson, Daniel Eth, and Sören Mindermann.