Introducing BenchBench

Existing benchmarks are quickly becoming saturated due to improvements in model performance. In response, BenchBench has emerged, which measures how well models can design their own benchmarks.

Each model reviews existing benchmark reports, then must create a new benchmark that is capable of overwhelming frontier models while still being actually solvable. When a model fails, it goes through an iterative process where the reason for failure is provided as feedback and it tries again.

As a result of testing, only GPT 5.2 was recorded as the sole winner, generating useful benchmarks that other models find difficult to solve. Other models, on the other hand, showed the following limitations.

• GPT-5.4: Built policy and governance scenarios, but they degenerated into simple checklist forms.
• GPT-5.5: Generated procedural rule tasks, but relied excessively on specific schemas or hidden labels.
• Gemini 3.1 Pro: Generated qualitatively differentiated tasks, but they were overly puzzle-like or fragile.
• Gemini 3.5 Flash: Excelled at commercial regulation-related problems, but the difficulty was low, so top models solved them easily.
• Claude Opus: Created elegant, competition-style problems, but they were too readable, making them easy to solve.