Building Agents That Don't Forget

LLMs are inherently stateless, so the memory felt in a conversation is created by re-pasting past records every time. This approach is sufficient for simple chat, but the limitations become immediately apparent once you build an actual Agent.

Without memory, the same information gets asked again, personalization disappears, and intermediate state during multi-step tasks gets cut off. Repeated mistakes also don't get corrected, knowledge doesn't accumulate across sessions, and when context overflows, the model tends to fill in gaps and produce hallucinations.

A longer context window is often thought of as the solution, but that alone isn't enough. As with the Lost in the middle phenomenon, accuracy drops when relevant information sits in the middle of a lengthened context, and since the system prompt, conversation history, retrieved documents, and output all share the same token budget, simply extending context has clear limits.

So we need to divide memory the way the human memory system does. Just as Sensory memory, working memory, and long-term memory each handle input retention, current thinking, and long-term storage, an Agent should also separate immediate working state from long-term memory. Long-term memory further splits into episodic, semantic, and procedural memory, and memory consolidation—where repeated events harden into general rules—becomes important.

The simplest Agent operates independently on every call. Attaching the entire conversation history as a Python list enables multi-turn conversations, but the list grows indefinitely, eventually hitting the context limit, and the memory disappears once the process ends.

The next step is storing memory on disk as Markdown files. This method is useful for prototyping since a person can directly open and edit it, but as the data grows to thousands of facts and hundreds of logs, keyword search alone struggles to find semantic synonyms or contextual connections.

Next comes vector search. Using embeddings to find semantic similarity connects expressions like "database" and "PostgreSQL," but the relationships linking multiple facts are still weak. Questions that require jumping two or three or more hops across people, projects, systems, and incidents are difficult to answer with flat vector search alone.

To solve this problem, what's needed is a memory layer that combines persistence, semantic understanding, and relational reasoning. Rather than directly wiring together a vector DB, graph DB, relational store, entity extractor, deduplication pipeline, and edge weighting system for this purpose, the article introduces an open-source knowledge engine called Cognee.

Cognee uses a three-store architecture.

• Relational store: provenance and access history
• Vector store: meaning and similarity
• Graph store: relationships between entities

The API is simplified into four asynchronous calls.

• add() to ingest documents
• cognify() to build the knowledge graph and embeddings
• memify() to refine memory
• search() to perform retrieval that includes reasoning

The default stack is composed of embedded, file-based components like SQLite + LanceDB + Kuzu, and the key point is that you can get started quickly while handling relational, vector, and graph memory all within a single system.