What's the largest software project AI can complete on its own

Epoch AI and METR have released MirrorCode, a long-horizon autonomous coding benchmark. Without access to the original source code or the internet, models must reimplement entire programs from scratch and produce outputs identical to the original on hidden end-to-end tests.

The benchmark includes 25 programs spanning diverse domains such as Unix utilities, data serialization/query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Some tasks were designed to take human engineers months to solve without AI, and a single run of the largest task cost 19 days and $2,600.

As a key example, Claude Opus 4.7 reimplemented gotree, a bioinformatics tool consisting of about 16,000 lines of Go code and more than 40 commands, in just 14 hours and $251. The researchers estimated this task would take a human engineer 2 to 17 weeks without AI.

The current regular leaderboard covers 30 tasks, implementing 15 Medium and Large-scale programs in each of Go and Ada. Each task is run three times with a budget of 10 billion tokens and 7 days. In the published results, Claude Fable 5 achieved a solve@100% rate of 64%, GPT-5.6 Sol 20%, GPT-5.4 16%, and GPT-5.5 10%.

However, since the target programs are open source, there's a possibility that the original code was included in the models' pretraining data. The researchers note that while their memorization checks don't fully explain the performance, the influence of data contamination cannot be ruled out. They have also released the benchmark scaffold and 22 of the 25 target programs.