How to Build Evals for Deep Agents

To improve agent performance, the key is not simply adding more tests, but designing Targeted Evals that directly measure behaviors that matter in real production environments. Adding tests indiscriminately can create the illusion of score improvements that are actually unrelated to real performance gains.

To build effective evals, the following approach is recommended.

• Define Target Behaviors: Determine the core behaviors an agent must perform (e.g., multi-file content search, chaining 5+ tool calls in a row) and curate evals that can verify them.
• Self-documenting: Add a docstring to each eval explaining how the corresponding capability is measured, and assign category tags like tool_use to enable grouped runs.
• Trace Analysis: Use tools like LangSmith to trace every eval run and analyze failure modes to update eval coverage.

Data sourcing is done through Dogfooding results from directly using internal agents, leveraging external benchmarks (Terminal Bench 2.0, BFCL, etc.), and manually writing unit tests for critical behaviors. In addition, eval reliability is improved by separating SDK unit/integration tests—such as system prompt delivery or routing—from Model Capability Evals that measure the model's own capabilities.