Google Sleep, Qwen3-8B Achieves 79.2% on AIME-24

• Google Research announced the Sleep framework, which stores knowledge in new low-rank MoE experts during continual learning without overwriting existing parameters
• Performance improved from 73.8% to 79.2% on AIME-24 and from 42.4% to 46.1% on HMMT-25, based on Qwen3-8B
• Training cost is 4 times higher compared to SFT, but reaching the same score requires 3.6 to 4.8 times more training time for SFT
• Official code and weights are not publicly available, and third-party reproductions have observed results differing from the original paper

Core Mechanism

• Knowledge accumulated in the KV cache during the Wake phase is transferred to new parameters in the Sleep phase to prevent catastrophic forgetting
• Combines on-policy distillation (GKD) and imitation learning (LTI) during knowledge seeding, with synaptic pruning to secure capacity
• Refines subtask performance in the SEAL-based Dreaming phase through random expert selection and gradient-based filtering

Experimental Results and Limitations

• Achieved 97% on CLINC, 92% on Banking, and 89% on DBpedia in incremental learning for the Llama3-8B class
• Recorded higher ChRF scores than ICL in continuous translation of unfamiliar languages, maintaining 91% at 10M tokens in long-context BABILong
• Theoretical foundations are included in the ICLR 2027 submission, but it is not currently in a deployable state