Replit Releases ViBench 2.0: Top Models Pass Only 49-65% of Long-Horizon Software Builds

Details

Replit released ViBench 2.0, a benchmark for long-horizon software development where agents build entire products one feature at a time, starting from an empty project.
GPT-6.1 Sol leads with a 65% pass@1 rate, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%; all three pass over 90% of individual test plans.
The benchmark includes 17 products, 241 feature requests, and 544 test plans, with eight products public on GitHub in the Harbor format.
Primary failure modes include schema errors where code references tables before creation and breaks at feature intersections, such as undo history conflicts or concurrency issues.
GPT-6.1 Sol costs approximately $12 per build, while GPT-6 Astra costs about $90; DeepSeek V4.1 Flash passes 77% of test plans for $2.50 but fails full builds.
Opus 5.5 serves as the grader, confirmed accurate by GPT-6.1 Sol re-grading 477 passes with 99.2% confirmation and no bias toward Claude models.
Scores improve across GPT generations, rising from 0% for GPT-5.4 to 53% for GPT-6.1 Sol on public products, with plan pass rates climbing from 42% to 95%.

Key point