How AI Agent Memory Works

LLMs are inherently stateless, so once a single call ends, they don't remember previous conversations. An agent is orchestration around the model, and memory is the loop that decides what to feed back into the next prompt.

The simplest approach is to re-insert the entire conversation history as-is. However, since the context window is limited, older turns must be dropped, and real-world systems use policies like summarization, retrieval, and hierarchical paging instead of FIFO.

Memory is usually divided into working memory and long-term memory. The former holds the current conversation and tool calls, while the latter is compressed into embeddings, stored, and retrieved when needed. Here, memory is not just storage but a lifecycle issue involving write, age, supersede, redact, and forget in sequence, where PII must be masked before storage and outdated facts should be left as superseded. Some designs even allow self-editing, where the memory modifies itself.

Embeddings convert the meaning of text into vectors, and queries are converted the same way, using cosine similarity and top-k to select the closest memories. The 2D diagram is just a simplified illustration for explanation—in practice, it works on the same principle in 1,536 or 3,072-dimensional spaces, like OpenAI's text embeddings.

Memory types are also organized into four categories:

• episodic memory: past conversations
• semantic memory: accumulated knowledge
• procedural memory: how to use tools
• working memory: the current prompt

In a practical RAG loop, at every turn the system takes user input, encodes it, and reassembles relevant memories and tool results into the context to generate a response.