UC Berkeley Publishes 'Harness Tax' Study: Up to 5x Cost Difference Between Harnesses

Researchers from UC Berkeley's Sky Computing Lab and Arena released study results analyzing the impact of coding agent harnesses (the software layer wrapping the model) on performance and cost. The core conclusion is that while task success rates remain nearly identical when changing harnesses, the cost to achieve the same quality varies by up to 5 times. The researchers named this phenomenon the 'Harness Tax'.

Experimental Design and Key Findings

The researchers conducted controlled experiments on the SWE-bench Lite and Terminal-Bench 2.0 benchmarks, combining 7 models (including Claude Fable 5 and GPT-5.6) with 3 harnesses (Claude Code, Codex CLI, and Pi).

• Cost Gap: For the GPT-5.6 Luna model, using the Claude Code harness ($0.152) compared to the Pi harness ($0.030) resulted in a 2.3 percentage point increase in success rate but approximately a 5-fold increase in cost.
• Success Rate Stability: Variations in success rates due to harness changes were limited.
• Pareto Frontier: The simple Pi harness, which provides only 4 tools (read/write/edit/bash), achieved the cost-performance Pareto frontier on both benchmarks.

Causes of Cost and Practical Implications

The primary cause of the cost difference stems from variations in context volume rather than the number of turns. Claude Code uses an initial context of 27,011 tokens on the first call, whereas Pi uses only 1,972 tokens. More complex harnesses lead to a rapid increase in initial context and loop length, resulting in higher token consumption.

The researchers argue that model evaluations should simultaneously compare cost and success rates across different harnesses. They also pointed out that even low-cost models can incur higher costs per solution if paired with inefficient harnesses, emphasizing that the role of the harness varies depending on task difficulty. While model intelligence is crucial for routine tasks, the structural contribution of the harness remains significant for complex problem-solving.