AI Researchers Discuss the Cumulative Nature of RSI and Limitations of Continual Learning

Dwarkesh Patel discussed the current state of AI research with Beren Millidge, CTO of Zyphra, John Schulman, Chief Scientist at Thinking Machines, and Charlie O'Neill, Head of Model Training at Baseten. The panel debated whether Recursive Self-Improvement (RSI) is theoretically a 'cumulative task' or a task in a non-stationary environment where the distribution shifts, similar to a law firm agent. Beren Millidge presented a scenario where, if meta-learning generalization and continual learning remain unsolved, AI could excel on benchmarks but be limited by the sim-to-real gap in actual environments. Dwarkesh Patel mentioned research related to pre-training progress driven by data and agreed that while applying mid-training and Reinforcement Learning (RL) on top of currently pre-trained base models appears similar to continual learning, it differs from true continual learning, which involves repeatedly applying small updates to the latest model without losing any existing knowledge.