Separating Noise from Signal in Coding Evaluations

Through a recent close investigation, OpenAI discovered that a significant number of tasks in the coding benchmark SWE-bench Pro are flawed. Analysis estimates that approximately 30% of all tasks are in a 'Broken' state that makes proper performance measurement impossible.

The major flaws discovered are classified into four categories:

• Overly strict tests: Enforcing detailed implementation specifics not stated in the prompt
• Underspecified prompts: Requirements demanded by hidden tests are missing from the prompt
• Low-coverage tests: Verifying only part of the requested functionality, allowing incomplete fixes to pass
• Misleading prompt: Leading the model to incorrect behavior or contradicting the test requirements

To accurately measure the model's actual capabilities, OpenAI built a data quality assurance (QA) pipeline. This pipeline operates by combining automated filtering with in-depth review using an Investigator-agent, as well as manual verification by experienced software engineers.

This result shows how difficult it is to build a benchmark that is both challenging and fair. OpenAI recommended that model developers carefully review benchmark results with this possibility of flaws in mind when interpreting them.