ProgramBench (5 min read)

ProgramBench is a benchmark that gives only an executable file and documentation and asks the agent to reimplement the original program. The agent cannot see the source code, cannot use the internet, and decompilation/analysis tools like objdump, strings, and hexdump are also blocked. In the end, the agent must handle everything itself, from language choice and architecture design to writing the entire codebase and build scripts.

The agent constraints are as follows.

• No access to source code
• No internet access
• Decompilation tools and binary analysis tools are blocked
• Full autonomy over language, structure, and build method

The evaluation scope covers 200 tasks and over 248,000 hidden behavioral tests. It ranges from small terminal utilities like jq and ripgrep to large projects like PHP, FFmpeg, and SQLite, and scores are split into resolved, meaning all tests passed, and almost resolved, meaning 95% or more passed.

The initial results are low. In the table, Claude Opus 4.7 was the highest with an almost resolved score of 3.0%, Opus 4.6 scored 2.5%, and Sonnet 4.6 scored 1.0%. GPT-5.4, Gemini 3.1 Pro, Gemini 3 Flash, Claude Haiku 4.5, GPT-5.4 mini, and GPT-5 mini all recorded resolved 0%.

The authors stated that they used mini-SWE-agent as a minimal common harness to avoid excessive harness tuning, and that some runs cost up to $5,000 based on Sonnet 4.5. ProgramBench is a benchmark that reveals how far current LLM agents still are from being able to design and implement a program from scratch.