AI BriefingBenchmarks for measuring the upper limit of AI performance are running out
Key point
As AI models advance rapidly, existing benchmarks are quickly becoming saturated, making it difficult to measure the ceiling of their performance.
Details
As the pace of AI model development accelerates, the problem of existing benchmarks becoming quickly saturated is worsening. Even a benchmark like GPQA, which was extremely challenging in early 2024, reached saturation in just one year.
METR's Time Horizon suite is also facing a crisis. Various long-duration tasks that AI could not previously perform are now being solved for the most part by the latest models such as Claude Opus 4.6 and GPT-5.3. As a result, it is becoming increasingly difficult to concretely define the upper limit of a model's performance.
Attempts to respond by developing new benchmarks are also running into limits in terms of cost and time.
- GPQA shocked academia at the time in 2024 by costing hundreds of thousands of dollars.
- Building expert baselines for METR's 50 long-duration tasks in 32-hour units requires more than 3,200 work-hours and at least over $1 million in cost.
Ultimately, due to the enormous cost and time required to create benchmarks, there is a significant risk of a vicious cycle repeating itself, in which AI conquers new benchmarks before they are even completed.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Anthropic Pacing to $100B ARR; Rumors of OpenAI Solving Navier-Stokes Spark ControversyOpenAI and Anthropic's RSI Strategy and the Uncertainty of AI Profitability
Study Finds Safety Training for GPT Models Transforms Rather Than Reduces Sexism