Better Harness: Hill-Climbing the Harness with Evals

Evals serve as a signal for agent harness engineering, much like model training data. The core idea is to build eval cases that clearly reveal desired behavior, and construct a hill-climbing loop that improves the harness one step at a time based on that signal.

How good evals are gathered also matters. Manually written cases, cases pulled from failures in production traces, and cases adapted from external datasets should all be used together, and each eval should be tagged with behavior labels such as tool selection and multi-step reasoning. This makes it possible to create category-based holdouts and run experiments on only the parts that are needed.

As with typical training systems, the goal is generalization. A small number of well-tagged evals is better than thousands of noisy cases, and a holdout set along with human review should be maintained together to prevent reward hacking, where the agent games the eval structure.

The Better-Harness loop proposed by the authors proceeds in the following order.

• Eval sourcing and tagging: Mix manual cases, production traces, and external datasets, and classify them into behavior categories.
• Train/holdout split: Split into an Optimization set and a Holdout set by category to reduce overfitting.
• Baseline measurement: Record baseline scores on both sets before making changes.
• Automated optimization: Diagnose root causes by examining traces, then apply harness changes—such as prompts or tool descriptions—one at a time.
• Validation: Check both whether new evals pass and whether previously passing cases regress.
• Human review: Have a human do a final check for overfitting or token waste that metrics miss.

Experimental results were also presented. When evaluating tool_selection and followup_quality on Claude-sonnet-4-6 and GLM-5, scores improved on both the optimization set and the holdout set after optimization compared to before. In particular, the problem of "asking unnecessarily many additional questions" and the problem of "incorrectly chaining new tools" were reduced, and specific instructions such as use reasonable defaults, respect already-fixed constraints, bound exploration before acting, and ask domain-defining questions first proved effective.

The key is the trace. Traces are the source that turns failures into evals, and comparing harness differences across versions explains what produced the improvement. Ultimately, a good agent isn't made by a good model alone—it comes from good evals, good traces, and a harness that is continuously refined based on them.