Building End-to-End Agent Evals with Harbor and Vercel

·2026.11.16 09:00

Key point

The open-source framework runs agents in containers to score success rates, efficiency, and cost, distinguishing between capability and consistency.

Details

  • Agents are now software users, requiring evals to measure task completion accuracy beyond simple functional tests.
  • The article proposes an agentic testing pyramid: deterministic tests, one-turn evals, and end-to-end (E2E) evals, with E2E evals being necessary to verify the final outcome of multi-step tasks.
  • E2E evals use metrics like Pass@k (capability) and Pass^k (reliability) to determine if an agent can perform a task or if it can be trusted to do so consistently; for high-stakes tasks like ordering coffee, Pass^k is the critical metric.
  • Harbor, an open-source framework from the creators of Terminal-Bench, runs these evals by executing agents in isolated containers, verifying final states against specific constraints, and recording full traces for analysis.
  • Trace analysis reveals specific failure modes, such as agents skipping price verification or failing to discover tools, allowing developers to fix product issues like adding budget constraints or improving discoverability.
  • Vercel provides the infrastructure stack for this workflow, including AI Gateway for model routing, Vercel Sandbox for concurrent trial execution, and Drives/Blob for storing traces and artifacts.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.