Memorilla Compresses LLM Memory into 16 Tokens for Better Long-Context Performance

Details

Memorilla is a memory system that encodes documents into reusable embeddings and retrieves query-specific information into a frozen LLM's input embedding space using only 16 memory tokens.
On the FactKG benchmark, Memorilla reaches 84.33% accuracy compared to 80.43% with full-context inference, while using 108x fewer tokens to represent source documents.
The system outperforms baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document, and knowledge-intensive question answering.
In a TextWorld agentic environment, Memorilla improves average success rate from 18.6% to 24.2% and reduces the agent's tendency to wander between already-explored rooms.
The approach facilitates continual learning in agentic systems where the document count increases throughout the trajectory.

Key point