SWE-bench Contamination Confirmed

OpenAI stated that SWE-bench Verified is no longer usable as a frontier model performance metric.

Over the past 6 months, scores rose from 74.9% to 80.9%, but OpenAI found that a significant portion of the remaining failures were not model limitations but dataset and test design problems.

There are two core issues.

• Test flaws: Among 138 problems reviewed by OpenAI, 59.4% had tests that wrongly failed correct answers or description mismatches.
  • 35.5% were narrow test cases that over-enforced implementation details.
  • 18.8% were wide test cases that required functionality not present in the problem description.
  • 5.1% were other issues.
• Training contamination: Frontier models reproduced details of gold patches or problem wording, leading OpenAI to judge they were likely already exposed to this during training.

To verify contamination, OpenAI used GPT-5 to conduct 15-turn probing against GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash Preview, and in some cases even the exact gold patch and implementation details were recovered.

OpenAI said it will now stop reporting SWE-bench Verified scores, and proposed SWE-bench Pro as an alternative. It also stated that it is building a new, uncontaminated code evaluation.