How to Build Evals for Deep Agents

To improve the performance of Deep Agents, evals (evaluations) are a core element that define and shape the agent's behavior. Simply adding a huge number of tests can create an illusion that fails to reflect real production behavior, so it's crucial above all to build targeted evals with a clear purpose.

To build effective evals, follow this process:

• Determine the core behaviors the agent should perform, and curate custom evals that can verify them.
• Add a docstring to each eval explaining how it's measured, and group them by assigning tags such as tool_use.
• Review traces in LangSmith to analyze failure modes and continuously update the eval coverage.

Secure data through dogfooding—using your own agent directly—leveraging external benchmarks like Terminal Bench 2.0 or BFCL, and writing manual tests for critical behaviors.

In particular, you should strictly separate the SDK's unit/integration tests from model capability evals that measure the model's capabilities. This removes unnecessary noise and lets you focus on actual model performance improvements.