AI BriefingAnalyzing the Cause of MMLU Benchmark Score Discrepancies
Key point
This analyzes the technical causes behind the differences in MMLU benchmark scores between the Open LLM Leaderboard and model papers.
Details
Following the recent release of the Falcon model, the community has been discussing how the MMLU scores shown on the Open LLM Leaderboard differ significantly from the figures published in the existing LLaMA paper.
To identify the cause of this discrepancy, the Hugging Face team compared and analyzed three different MMLU implementations:
- EleutherAI Harness: the implementation used by the Open LLM Leaderboard
- Stanford HELM: the implementation used in Stanford's comprehensive evaluation benchmark
- Original Implementation: the original implementation developed by the UC Berkeley team
The analysis confirmed that even when using the same MMLU dataset, results vary depending on which evaluation code (implementation) is used. This stems not from differences in the models' own performance, but from differences in the library and code structure used to run the benchmark.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Privatemode Benchmarks GLM-5.3-Flash as Vision-Capable Decision Model with Parity to Jev but Higher Cost
Meta Releases 'WearableQA' Health Reasoning Benchmark Based on Real Wearable DataAPL verifies MMLU score comparability; returns 'incomparable' if no applicable bridge exists