AI BriefingGPT 5.5, First ProgramBench Solve
Key point
In ProgramBench's first full solve, GPT-5.5 outperformed Opus 4.7.
Details
ProgramBench is a benchmark that gives only a compiled binary and documentation, requiring the source to be reimplemented from scratch. In a new update, GPT 5.5 (high) and GPT 5.5 (xhigh) fully solved cmatrix, producing the first success cases.
GPT 5.5 (xhigh) outperformed Claude Opus 4.7 (xhigh) on every key metric.
- Full solve: 0.5% vs 0%
- 95%+ solve: 13.5% vs 4.5%
Individual runs also showed a large efficiency gap.
- GPT 5.5 (high): C implementation, 34 API calls, $3.17
- GPT 5.5 (xhigh): Python implementation, 40 API calls, $4.84
- Claude Opus 4.7 (xhigh): 178 API calls, $10.74, 19 failures
The authors explained that GPT-5.5 batched actions into larger chunks, such as chaining commands with &&, resulting in fewer agent steps and higher token efficiency.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Privatemode Benchmarks GLM-5.3-Flash as Vision-Capable Decision Model with Parity to Jev but Higher Cost
Taste-Bench: Frontier LLMs Achieve 59.7% Accuracy on Decision Forking TasksTechnical Rebuttal of TypeSafe Jev Model's 'Calibrated Probabilities' Claim