ProgramBench Benchmark Released

Facebook Research has released ProgramBench. It is a benchmark that evaluates whether the original program can be reimplemented using only the executable file and README/usage file.

• It consists of 200 tasks, and agents handle everything from language choice to designing abstraction layers to configuring the overall architecture.
• Under conditions of no internet access and no decompilation, scoring is based solely on black-box execution results.
• About $50,000 was spent to generate 6 million lines of behavioral tests, which were then filtered by quality criteria.

Along with the official site, GitHub, Hugging Face, and a Docker image have been released, and evaluation can be started right away with pip install programbench && programbench eval <your submission>.

The currently released results are centered on closed models, and it was explained that open-source models, being more tuned to SWE-bench style tasks, struggle more with the new benchmark. A submission portal is also planned to open soon.