HOLA: Overcoming the Memory Limits of Linear Attention

Linear-attention and State Space Models (SSM) increase memory efficiency by using a fixed-size recurrent state, but as information is compressed, a lossy memory problem occurs where earlier data gets overwritten.

The newly proposed HOLA (Hippocampal Linear Attention), inspired by Complementary Learning Systems, solves this problem by adding a bounded exact KV cache alongside the existing compressed memory, the delta-rule state.

Key Features and Results:

• Semi-parametric test-time memory: The state models structures that can be linearly compressed, while the cache stores important associative relationships that are hard to fit into the state.
• Efficient cache management: Without a learned eviction module, tokens with large prediction residuals are selected and kept in the cache.
• Performance improvement: On a 340M-parameter model, Wikitext perplexity improved by 16.1%, from 27.32 to 22.92, surpassing Transformer++ performance.
• Long-context retention: In the Needle-in-a-haystack test of the RULER benchmark, it maintains very strong retrieval performance up to 32k tokens, 16x the training length.