Language Models Need Sleep Too: A Self-Correction and Memory Consolidation Training Method

Existing LLMs excel at immediate prediction or in-context learning, but lack the ability to continuously learn or effectively transfer short-term knowledge into long-term parameters.

To address this, a 'Sleep' paradigm that mimics the human learning process is introduced. This approach is designed to let the model refine short-term memory into stable long-term knowledge and recursively improve itself.

The 'Sleep' phase consists of two main processes:

• Memory Consolidation: A bottom-up distillation process called 'Knowledge Seeding.' It distills the memory of a smaller model into a larger network, preserving knowledge while securing broader capacity. For this, a new Generalized Distillation process is used, combining on-policy distillation with RL (reinforcement learning)-based imitation learning.
• Dreaming: The stage where the model improves its own performance. Using RL, it generates a synthetic data curriculum, reviewing new knowledge and refining existing abilities without human supervision.

Experimental results demonstrate the effectiveness of this approach on long-horizon continual learning, knowledge consolidation, and few-shot generalization tasks.