FrontierHarness Eval v1.1: Model-Harness Pairing Determines Cost-Performance
·2026.11.17 09:00
Key point
Across 22 harnesses and 5 models, the optimal harness for cost-efficiency differs per model, with model choice explaining 92.8% of cost variance.
Details
- FrontierHarness Eval v1.1, a collaboration between Runta and Baseten, evaluates 22 harnesses across 5 models using 62 tasks and 27,280 agent runs (83B tokens), a 76x expansion from v1.0.
- The optimal harness for cost-performance (lowest cost per pass) varies by model: Codex CLI for GLM-5.3, Pi for GLM-5.3 Flash, ZCode for DeepSeek V4.1 Flash 1017, Oh My Pi for Muse, and Alma for Kimi-K3.
- Model selection accounts for 92.8% of cost variance, while harness choice accounts for 2.3%; with a fixed model, the most expensive harness costs 1.9–4.9x more per pass than the cheapest.
- Steps (agent trajectory entries) are the primary cost driver, showing a +0.51 rank correlation with cost per pass, whereas prompt size, tool count, and cache hit rate show negligible correlation.
- 64.5% of total inference cost was spent on non-passing trials (52.7% failures, 11.8% timeouts), and higher cost does not guarantee higher pass rates (rank correlation -0.39 to +0.22).
- Best-value harnesses typically use a single shell tool for 61–93% of tool calls, maintain high cache hit rates (91.1–94.0%), and minimize subagent use and summarization.
- Limitations include the use of open models only via a single provider (Baseten); closed models and different providers may alter cost rankings and results.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.