Independent analysis finds AI labs report benchmark scores 6.4% higher than third-party tests

·2026.10.19 18:47

Key point

A comparison of 611 matched results across 180 models reveals that self-reported scores are consistently higher than independent measurements.

AI labs report benchmark scores 6.4% higher on average compared to independent testers for the same models and tests.
The analysis is based on 1,629 published benchmark results, matching 611 comparisons across 180 models and 103 benchmarks.
The project whatmodel.sh is open source, with the dataset released under CC BY-SA 4.0 and code under the MIT license.
Sources and licenses are recorded with links provided on each model's page, and an interactive chart is available at whatmodel.sh/claims.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.