A Checklist for Preparing Agent Evals

Agent evaluation is fundamentally different from traditional software testing. Before building complex automated systems, it's important to start with a Simple Eval to check whether core tasks are being completed.

The essential preparation steps that must be taken before building evaluation infrastructure are as follows.

• Manually review real traces: Before building infrastructure, read through 20-50 real agent traces yourself to identify failure patterns.
• Define clear success criteria: Standards must be specific enough that two experts would reach the same pass/fail judgment.
• Separate capability evals from regression evals: Capability Evals, which test new features, and Regression Evals, which confirm existing features are maintained, should be managed separately.
• Build the ability to analyze failure causes: If you can't clearly explain why something failed, you're not yet at the stage to build automated evals. 80% of your effort should go into collecting failure cases and building a Failure Taxonomy together with domain experts.

The solution differs depending on the cause of failure. If it's a prompt problem, fix the instructions; if it's a tool design problem, redesign the parameters or interface; and if it's a limitation of the model itself, you need to consider a different approach.