MirrorCode (8-minute read)

MirrorCode is a long-horizon coding benchmark where AI models reimplement entire programs from scratch without access to the original source code. The generated implementations must produce outputs that exactly match the originals in end-to-end tests, including private tests.

The benchmark includes 25 programs across diverse domains such as Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.

MirrorCode provides a significantly larger inference budget than existing software engineering benchmarks. One of the largest tasks cost $2,600 for a single run, with the AI working for 19 days without human intervention.

The tasks are difficult enough to take human engineers months, but are designed to be fairly solvable with sufficient information. Additionally, access to the internet and the original codebase is blocked, and private tests that are invisible during development are used to prevent cheating and the creation of simple output lookup tables.

Claude Opus 4.7 reimplemented the bioinformatics tool gotree, which includes approximately 16,000 lines of Go code and over 40 commands, in 14 hours for $251. The researchers estimated that a human engineer without AI assistance would take 2 to 17 weeks to complete this task.

However, since the target programs are open source, the possibility of pre-training data contamination remains. Some tasks succeeded even after passing memorization checks, while tasks where evidence of memorization was confirmed failed. While performance is not explained solely by memorization, its influence cannot be completely ruled out.

The current leaderboard targets 30 tasks involving the implementation of 15 programs from the Medium and Large buckets in Go and Ada respectively. Each task is run three times with a budget of 10 billion tokens and 7 days per attempt. The published solve@100% rates are 64% for Claude Fable 5, 20% for GPT-5.6 Sol, 16% for GPT-5.4, and 10% for GPT-5.5.