Evaluating Deep Agents: What We Learned

LangChain recently launched four applications—DeepAgents CLI, LangSmith Assist, Personal Email Assistant, and Agent Builder—and in the process built out an Evaluation process for Deep Agents.

While traditional LLM evaluation relies on the same dataset and evaluator, Deep Agents require unique success criteria for each data point. This is because it's not enough for an agent to simply produce a good final answer—its Trajectory and State must also be verified as correct.

The 5 core patterns for effective evaluation are as follows:

• Custom test logic: Apply unique success criteria and validation logic for each data point
• Single-step execution: Verify the agent's decision-making ability in specific scenarios while reducing token consumption
• Full agent turns: Assertion testing on the agent's End state
• Multiple agent turns: Simulate real user interactions while keeping the agent from straying off the intended path
• Environment setup: Build clean, reproducible test environments

For example, when testing a calendar agent that remembers user preferences, it's not enough to simply check the answer—you need to verify from multiple angles, using methods like LLM-as-a-judge, whether the agent correctly modified a specific file (memories.md) and whether its contents match the user's request.