GPT-5.5 low vs medium vs high vs xhigh: The Reasoning Curve Seen Across 26 Real Tasks in an Open-Source Repository

This is the result of an experiment that split GPT-5.5 Codex's reasoning effort settings into low, medium, high, xhigh and performed 26 tasks on a real open-source repository (GraphQL-go-tools).

The key finding is that as reasoning effort increases, far more dramatic improvements appear in Semantic Equivalence and code review pass rate than in simple test pass rate.

Performance comparison by setting

• low & medium: Test pass rate is identical at 21/26, but the medium setting models the repository's domain semantics better, noticeably improving equivalence and review quality.
• high (optimal balance): All metrics improved compared to medium, and it shows the largest gain in performance relative to cost ($4.49), making it the practical default setting.
• xhigh (high cost, high quality): Quality is highest, but average cost jumps sharply to $9.77, execution time lengthens, and there's an increased footprint risk of modifying more code than human-made patches.

Conclusion and implications Existing benchmarks that simply measure whether tests pass (binary results) fail to properly reflect an AI agent's actual engineering capability. A dedicated evaluation system (harness) capable of measuring whether an agent-generated patch aligns with human intent and is maintainable is essential.