AI BriefingProgramBench: Can Language Models Rebuild Programs from Scratch
Key point
ProgramBench evaluated LLMs' ability to reconstruct entire programs at scale.
Details
ProgramBench is a benchmark that provides only a program and its documentation, and requires reimplementing a codebase from scratch to match the behavior of a reference executable.
It is designed to measure comprehensive development ability, including software architecture decisions, rather than a single bug fix or a single feature implementation.
Evaluation was conducted with end-to-end behavioral tests created via agent-driven fuzzing, without prespecifying the implementation structure.
- Number of tasks: 200
- Scope: from small CLI tools to FFmpeg, SQLite, PHP interpreter
- Models evaluated: 9 LMs
- Result: no model fully solved any task
- Best performance: only 3% of all tasks passed 95% of tests
Unlike human-written code, models favored single-file, monolithic implementations, showing large structural differences from the reference implementations.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Toss Doctor V2 Released: Maximizing QA Efficiency with LLM-based Automated Testing and Self-healing