Effective Context Engineering for AI Agents

Context engineering goes beyond writing good prompts—it's about designing how to select and maintain the entire set of tokens that go into an LLM's reasoning process. Elements like the system prompt, tools, message history, external data, and MCP must be handled together for an agent to reliably produce the desired behavior.

The core premise is that context is a finite resource. As token count grows, the model's attention becomes more diffuse, and over long contexts, information retrieval and long-range reasoning performance can gradually degrade. Good context design, therefore, isn't about packing in more information—it's about keeping only the smallest, most refined set of high-signal tokens needed to achieve the goal.

The system prompt should avoid both overly complex hardcoding and overly abstract instructions, guiding behavior at an appropriate altitude. Clear section boundaries, structuring via XML tags or Markdown headers, and a minimal set of examples based on observed failure patterns all help. Rather than piling on edge cases, examples are better organized around canonical examples that well represent the expected behavior.

Tools are the channel through which an agent interacts with its environment and pulls in new context as needed. Too many tools, or tools with overlapping functionality, make selection itself difficult and lead to wasted context from incorrect tool use. Each tool should be self-contained and error-resistant, and its input parameters should be unambiguous.

Recently, as pre-embedding-based retrieval alone has proven insufficient, just-in-time context—dynamically fetching needed information at execution time—has been gaining traction. In this approach, only lightweight identifiers such as file paths, saved queries, or web links are kept, while the actual data is loaded via tools only when needed. As seen in Anthropic's Claude Code, a hybrid strategy—loading some context upfront and letting the rest be explored through tools like glob, grep, and Bash—is presented as a practical solution.

For long-running tasks, in order to maintain goals that exceed the context window, it's important to design for a mix of incremental discovery and external storage rather than relying entirely on internal memory. Ultimately, an agent's performance depends not just on the model, but on the context operation skill—determining what to remember and what to retrieve.