Mercor Benchmark Map Reveals 90% of US Working Hours Lack Adequate AI Benchmark Coverage

Benchmark Coverage Gaps

Mercor’s new Benchmark Map indicates that 90% of American working hours are spent on tasks with little or no coverage from existing AI benchmarks. The study analyzed 29 benchmarks against the US Department of Labor’s O*NET taxonomy, which defines 923 occupations and 18,797 tasks.

Key findings include:

• Only 208 of 923 occupations have at least one task covered by 5 or more benchmark prompts.
• 6 of 22 job families have no occupations with tasks meeting this coverage threshold.
• Coverage is heavily skewed toward high-wage, high-AI-exposure roles like software engineering and financial analysis, while middle-income knowledge work and physical service jobs remain underrepresented.

Methodology and Bias

The research prioritized benchmarks that are publicly available, agentic, economically realistic, and unsaturated. Using an LLM judge to map benchmark prompts to O*NET tasks, the team found that tasks rated as "AI-exposed" are 6x more likely to be benchmarked than those rated "unexposed." Additionally, tasks from occupations with larger wage bills are significantly overrepresented, creating blind spots in understanding how AI performs across the broader economy.

Implications for Model Evaluation

The study demonstrates that simple averages of benchmark scores can misrepresent real-world performance. By applying O*NET weighting to Mercor’s APEX benchmarks for four occupations (investment banking, law, accounting, software engineering), the researchers compared simple averages against task-weighted scores. While the differences were minor in this subset, the approach aims to correct for the over-benchmarking of specific tasks. However, significant challenges remain, including low statistical power for many tasks, varying task difficulty, and the difficulty of measuring AI augmentation versus automation.