AI evals are becoming the new compute bottleneck

Static benchmarks can still be compressed by as much as 100–200x, but as you move toward agent benchmarks and training-in-the-loop evals, the cost savings shrink sharply.

• HAL spent about $40,000 on 21,730 rollouts across 9 models and 9 benchmarks, with a single GAIA run reaching up to $2,829 for frontier models.
• Exgentic's $22,000 sweep showed a 33x cost difference on the same task, revealing that scaffold choice is the primary cost driver.
• Even within the same benchmark, costs can diverge by 3–4 orders of magnitude depending on the model × scaffold × token budget combination.
• The Well requires 960 H100-hours (about $2,400) to evaluate a single new architecture, and 3,840 H100-hours (about $9,600) for a full 4-baseline sweep.
• Benchmarks closer to real-world tasks, like PaperBench, MLE-Bench, ResearchGym, and RE-Bench, see evaluation costs soar into the thousands to tens of thousands of dollars.

Securing reliability through repeated runs drives costs even higher. As benchmarks move closer to real-world tasks, evaluation itself becomes a new compute bottleneck.