How AI Agent Benchmarks Were Broken and What Comes Next

Structural vulnerabilities were discovered in 8 major AI agent benchmarks that allow high scores to be achieved without actually solving the intended problems.

The research team used an automated scanning agent to examine SWE-bench, WebArena, OSWorld, GAIA, and others, finding paths that exploit loopholes in the scoring logic to produce scores close to 100%.

The core issue is that the evaluation method itself could be manipulated, rather than reflecting the model's actual ability. If a structure allows scores to rise regardless of whether the task is genuinely performed, that benchmark cannot be trusted as a standard for comparing agent performance.

Ultimately, this finding reads as a warning that future evaluation design needs not only benchmark accuracy but also anti-cheat mechanisms, stronger verification logic, and reflection of real tasks.