Benchmark distortion caused by gold-like answers

When the reducer used in Maestro's coding agent evaluation showed unexpectedly high performance, the initial suspicion was SWE-bench Verified data contamination. However, when the same bias was reproduced on the more recent SWE-rebench dataset, it became clear that the core issue was not simple memorization but a tendency to favor patches that look like the gold answer.

This reducer is an LLM judge that selects one candidate patch among those produced by multiple agents. The authors built a basic reducer using Claude Opus 4.5 with a short prompt instructing it to "look only at correctness, not code quality or style," and evaluated it on 400 reducer-relevant examples.

To test the suspicion, they split the examples into two types:

• positive vs. negative: a positive candidate and a negative candidate arranged randomly
• gold vs. negative: the same negative candidate paired with the actual gold patch

As a result, the reducer showed much higher accuracy on gold vs. negative. However, when the same experiment was repeated on 169 reducer-relevant examples from SWE-rebench, the same gap appeared, leading to the conclusion that this phenomenon is not necessarily the result of memorizing specific answers.

The authors' interpretation is as follows. The model likely did not memorize the content of the patches, but rather learned surface-level characteristics typical of gold patches, such as minimality, clarity, and focused change. In fact, in the examples, even though the correct patches created by agents modified about 50% more lines than the gold patches, the reducer tended to prefer cleaner, smaller changes.

In other words, in evaluations like SWE-bench, "cleanliness" is not the same as correctness. Since the criterion is passing hidden tests, a clean but wrong patch can be rated higher than a messy but correct patch.

To mitigate this, the authors refined the reducer prompt multiple times, adding specific priority rules that prioritize substantive correctness over style. In conclusion, this shows that distortion factors in coding benchmarks must include not only data contamination but also the inherent bias of evaluation models toward favoring answers that 'look like gold.'