Google Research introduces sleep framework for LLMs to enable continual learning and retain 10M tokens

• Google Research proposes a sleep framework that allows LLMs to continuously integrate new facts and improve reasoning while mitigating catastrophic forgetting.
• The method achieves near-perfect long-context scaling up to 10M tokens on the BABILong benchmark, outperforming GPT-4 and Titans.
• Sleep-enhanced models reach 79.2% accuracy on AIME 24/25, 69.0% on AIME 25, and 46.1% on HMMT 25, surpassing baselines like SFT and GRPO.
• In continual translation tasks for unseen languages like Manchu and Kalamang, the framework maintains near-zero performance drop, whereas standard baselines suffer severe catastrophic forgetting.

Mechanism

• Active (wake) phase stores temporary information in short-term memory; sleep phase pauses input processing for offline consolidation.
• Memory consolidation uses Mixture-of-Experts (MoE) and Hope architecture to unlock low-rank experts, transferring knowledge via Generalized Knowledge Distillation (GKD) and reinforcement learning.
• Dreaming phase uses random expert selection and gradient-based data selection to generate synthetic training samples for self-improvement.

Theoretical Basis

• Knowledge seeding is framed as a low-rank compression problem that preserves on-policy behavior and semantic qualities.
• Random expert activation during dreaming broadens gradient directions, improving the geometry of learning and stabilizing optimization.