AI BriefingRoot Cause of DROP Benchmark Scoring Error Identified
Key point
The abnormally low model scores observed on the DROP benchmark in the Open LLM Leaderboard were found to be caused by a normalization error in the evaluation metric.
Details
In the DROP (Discrete Reasoning Over Paragraphs) benchmark added to the Open LLM Leaderboard, an anomaly was discovered in which most models recorded low F1 scores of under 10 out of 100.
Upon investigation, it was revealed that this was not a model performance issue but rather a technical error occurring during the normalization process of the evaluation metric.
The main causes are as follows:
- Number normalization failure: After a model generated the correct number, if a special character such as a newline (\n) followed immediately after the number instead of a space, the normalization step failed to recognize it as a number.
- Data mismatch: As unnormalized strings were compared in bag-of-words (BOW) form, cases where the model had actually answered correctly were processed as not matching the answer data.
This analysis was conducted in collaboration with Zeno, and it suggests that benchmark scores can distort a model's actual performance.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Privatemode Benchmarks GLM-5.3-Flash as Vision-Capable Decision Model with Parity to Jev but Higher CostTechnical Rebuttal of TypeSafe Jev Model's 'Calibrated Probabilities' Claim