AI BriefingCognition releases FrontierCode 2.0, a coding benchmark where top models score only 45%
·2026.10.28 02:00
Key point
The new benchmark features tasks 67% larger than previous versions and reveals that models struggle with unstated constraints and thoroughness, with the best model achieving a 45.5% score.
Details
- Cognition introduced FrontierCode 2.0, a new coding benchmark designed to evaluate agentic software engineering capabilities, specifically focusing on user intent inference, unstated constraints, and long-term code consequences.
- The benchmark is far from saturated; the top-performing model, Claude Opus 5.5, scored 45.5%, while GPT-6 Astra scored 42.8% and Grok 4.7 scored 28.6%.
- FrontierCode 2.0 tasks are significantly larger and more complex than FrontierCode 1.1, with reference solutions modifying 67% more files and 89% more lines of code in codebases that are 25% larger.
- The benchmark places heavy emphasis on "thoroughness" criteria, which grew by 3x compared to the previous version, testing whether models fix all instances of bugs and match existing support patterns rather than just addressing the explicit prompt.
- Higher reasoning effort does not always improve scores; models often fail scope criteria at max effort by making unnecessary changes or breaking public APIs, such as deleting a public API file in the PostHog SDK to fix a race condition.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Jack Clark tells IMF AI Singularity has begun, citing 98% score on unpublished physics problems