Language Models Need Sleep Too

Transformer-based LLMs suffer performance degradation on long-horizon tasks due to the scalability problems of the attention mechanism.

To address this, a 'sleep-like consolidation mechanism' is proposed, which converts recent context into persistent fast weights and then clears the KV cache.

• How it works: During the 'sleep' period, the model performs $N$ offline recurrent passes over the accumulated context, updating the fast weights of SSM(State-Space Model) blocks according to learned rules.
• Advantages: By offloading additional computation to the sleep phase, the model can effectively handle long contexts while preserving inference-time latency.

Experimental results show that the proposed model outperformed existing Transformer and SSM-attention hybrid models on complex tasks such as cellular automata, multi-hop graph search, and mathematical reasoning. In particular, increasing the sleep time ($N$) further improved performance on cases requiring deep reasoning.