Recursive Language Models, Explained Simply

Even large LLMs that boast 1M token context windows fail to properly reason over the information needed to answer questions in long inputs like 200-page documents or 50K token documents. The problem isn't length itself, but rather context rot, where reasoning quality collapses as the number of tokens increases.

The needle-in-a-haystack benchmark shows retrieval — finding a hidden sentence — well, but it doesn't reveal reasoning, such as counting or classifying thousands of items. The same pattern shows up as increased repetition and disorganized responses in long Claude Code sessions or long ChatGPT conversations.

The Recursive Language Models (RLMs) proposed by MIT researchers don't put the entire context into the prompt at once. Instead, they separate it out into a ctx variable in runtime memory. The model receives only the question, then explores ctx as a tool, much like working with a dataframe in Jupyter.

There are four core tools.

• Peek: Skim the beginning to grasp the document's structure.
• Grep: Use regex to extract only the relevant lines.
• Partition: Split the context into smaller chunks.
• Recursive call: Call itself again on the needed piece.

This approach differs from agent workflows predetermined by humans. The model itself decides what to look at first, when to split, and in what order to dig deeper. Even with the same input, the exploration strategy adapts to the task, and the context is continuously kept small.

For example, consider the task of counting billing questions from 3 specific users among 5,000 customer support tickets. An RLM would first grasp the structure, then grep only for the target users to narrow it down to around 50 lines. It would then recursively classify each item as billing/other, keeping the root model's context small.

The advantages of this approach are maintaining accuracy even on long documents, the ability to handle inputs on the scale of 10M tokens by splitting context into even smaller pieces, interpretability of the exploration process, and lower cost than a single large call. The key lies in treating context not as something to be summarized, but as data to be explored programmatically.