Why SWE-bench Verified No Longer Measures Frontier Coding Ability

OpenAI pointed out that SWE-bench Verified no longer properly measures the coding capability of frontier models.

Recent top performance has climbed from 74.9% → 80.9%, but it has become difficult to distinguish whether remaining failures stem from model limitations or benchmark flaws.

After reviewing 138 audited problems, significant flaws in test design or problem descriptions were confirmed in 59.4% of them.

The main issues are as follows.

• 35.5%: overly restrictive tests that force a specific implementation
• 18.8%: excessive tests that demand additional functionality not present in the spec
• Some cases had a structure that failed even functionally correct solutions

Also, due to the nature of evaluations based on public data and codebases, training data contamination was hard to avoid, and some models nearly reproduced the gold patch from just a task ID or a short description.

Accordingly, OpenAI has stopped reporting SWE-bench Verified scores and is shifting its evaluation axis toward SWE-bench Pro, which is less affected by contamination, and private benchmarks.