Agent Judge: Solving the Long-Context Evaluation Problem for Production Agents

Most teams evaluate agent performance in a simple way: passing the user query, the agent's final output, and evaluation criteria to an LLM judge. However, with the emergence of Long-horizon agents that handle broad and lengthy tasks, this existing approach is hitting its limits.

The main reasons existing LLM judges fail are as follows.

• Long trajectories: When a coding agent performs hundreds of tool calls, the entire process cannot fit into a single context window.
• Inability to verify stateful changes: When an agent updates a CRM or changes AWS settings, the judge cannot directly check the actual state of external systems like Google Calendar, CRM, AWS, or GitHub.

These limitations reduce the effectiveness of automated evaluation, causing agent errors to go undetected and ultimately forcing reliance on manual review. Solving this requires an Agentic judge equipped with retrieval, verification, and adaptation capabilities.