Vercel demonstrates how to build end-to-end agent evals using Harbor and AI Gateway

Details

Evals measure whether agents complete tasks correctly, distinct from deterministic tests that only check if code runs; they are essential because LLM outputs vary and single manual tests are insufficient.
The article proposes an agentic testing pyramid: deterministic tests (0 LLM calls), one-turn evals (1 call), and E2E evals (2+ calls), with E2E evals being the only tier that validates the full sequence of steps for user success.
A worked example using the sfbeans CLI and Harbor framework demonstrates that a 60% success rate can yield a 99% Pass@5 but only an 8% Pass^5, highlighting that reliability metrics must match the cost of failure for the user.
Trace analysis is critical for identifying specific failure modes, such as agents skipping price quotes or failing to discover tools, which allows for targeted fixes like adding budget constraints to the CLI.
The workflow involves defining a goal, selecting metrics (Success, Pass@k, Pass^k, Efficiency), designing tasks with verifiers, running concurrent trials via Vercel Sandbox, and analyzing trajectories to iterate on the product.

Key point