Comparison of 9 AI Agent Harnesses Based on the Same Model Reveals Up to 17x Cost Difference Per Task

The results of the FrontierHarness v1.0 evaluation showed that even with the same model and environment, the cost per task varies by up to 17x depending on the implementation of the AI agent harness. The evaluation targeted nine major harnesses, including Codex, Claude Code, and OpenCode, performing software engineering and terminal-based tasks.

The Gap Between Cost and Performance

• Cost Efficiency Gap: The most efficient harness recorded a cost of less than $1 per task, while inefficient harnesses saw costs rise to $18.34.
• Success Rate and Cost Are Not Proportional: Claude Code successfully completed 19 tasks but incurred high costs ($18.34). In contrast, some harnesses achieved similar success rates at lower costs.
• The Cache Hit Trap: A high cache hit rate does not necessarily lead to cost savings. Failures spanning 300 turns can consume more cost than short cache misses.

Evaluation Methodology and Limitations

• Fair Comparison Environment: All harnesses were restored from a golden checkpoint with identical vCPU, memory, and disk states within the Runta agent runtime before execution.
• Scope of Application: This evaluation is specialized for software engineering and terminal tasks and may not generalize to other domains of knowledge work.
• Data Adjustment: For OpenCode, only 15 successful tasks were aggregated by excluding failed attempts; however, including failure costs increases the cost per task to $3.24.