Gaming the Most Watched AI Agent Benchmarks

After automatically scanning 8 major AI agent benchmarks, UC Berkeley researchers confirmed that virtually all benchmarks are exploitable. The core message is that the scoreboards aren't measuring capability — they're measuring loopholes in the evaluation pipeline.

The targets were SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench. The researchers' agent built exploits that actually worked on each benchmark, passed the official evaluators as-is, and achieved near-perfect scores.

Key cases include the following.

• Terminal-Bench: Achieved 89/89, 100% using wrappers/trojans that hijacked system binaries like curl, uvx, pip, and python.
• SWE-bench Verified / Pro: Achieved 500/500 and 731/731 by manipulating test results via pytest hooks in conftest.py, unittest monkey patches in Django, and parser overwrites.
• WebArena: Since Chromium could read local config files via file:// URLs, the answers inside task configs were directly exposed. Combined with DOM injection and prompt injection, 812 tasks were effectively exposed.
• FieldWorkArena: The verification function only checked whether the last message was from the assistant, not the content of the answer. This meant 890/890, 100% was achievable with just an output like {}.
• OSWorld: Achieved over 73% easily by directly downloading publicly available gold files for the evaluator to compare against, or by directly matching the system state. Some even allowed code execution on the evaluation machine via use of eval().
• GAIA: Because of publicly available answers and an overly loose normalize_str comparison, the correct answers effectively functioned as a lookup table. A comma-handling bug and the filter meant to prevent 100% scores could also be bypassed.

Beyond listing individual bugs, the piece strongly warns that benchmark scores are disconnected from actual capability. The conclusion is that the design of the evaluation environment — permission separation, answer concealment, and verification logic — matters far more than the model's reasoning ability.