Deep Agents Benchmark Methodology

During the development of Deep Agents, various decisions had to be made regarding prompt design, tool selection, middleware configuration, and more, requiring a robust evaluation system to support them. The team moved away from unit-test-style methods and shifted toward end-to-end evaluation, adopting the Harbor framework as the evaluation runner.

Harbor performs evaluation by taking three inputs: an agent, a dataset, and a sandbox. Each task consists of a Dockerfile-based environment, markdown instructions, and an evaluation script (test.sh). Unlike simple LLM evaluation, the environment in which the agent runs matters, and scoring is done via script not just on the final response but also on the artifacts produced.

Three benchmarks are currently in operation:

• Harbor-Index: 82 tasks curated from 54 benchmarks, covering software engineering, search, data analysis, and more
• τ³-bench: multi-turn conversation evaluation, 30 tasks
• ContextBench: evaluation of search and composition abilities, 30 tasks

Practical operating principles were also shared. Due to non-deterministic characteristics, each task is run multiple times, and a separate 'lite' benchmark that is 8x faster and 6x cheaper than the full version is maintained for rapid iteration. A deterministic unit test suite that verifies specific harness behaviors is also run in parallel.

Using this system, the team is preparing the Deep Agents 0.7 release, and is considering removing the todo-list middleware and simplifying the system prompt. Benchmark results confirmed that reducing the system prompt improves performance on Harbor-Index.