OpenAI Recommends Halting SWE-bench Reporting; Solution is 'Evaluator-Led Verification'

In February, OpenAI recommended halting the reporting of the SWE-bench Verified benchmark, citing that all frontier models can reproduce human-written patches or problem statements for some SWE-bench Verified tasks. This measure reflects the phenomenon where scores plateau regardless of actual improvements in model capabilities.

Existing 'data decontamination' reports cannot resolve benchmark contamination issues due to three fundamental limitations:

• Limitations of self-verification: Outsiders cannot access training datasets, so they cannot reproduce the search process.
• Non-disclosure: Datasets contain lists of copyrighted works, posing legal risks if disclosed.
• Incompleteness of matching: Models can learn correct answers through forms that do not overlap in n-grams, such as paraphrasing, forum posts, GitHub code, and synthetic data.

Accordingly, the author proposes a method where evaluators control the tests. This involves not providing labels to submitters, running evaluations without network access, and requiring evaluators to directly build code from specified commits to reproduce scores. Additionally, reliability is ensured by generating test data after the submission deadline.

The author released a small-scale framework applying these principles but acknowledged that some limitations remain, such as preventing information leakage through repeated submissions to hidden test sets.