Real-SWE Benchmark Released: AI Coding Agent Evaluation Results Based on Proprietary Enterprise Codebases

The Real-SWE benchmark is an AI coding agent evaluation tool based on proprietary real-world enterprise codebases that assesses proprietary system exploration, business impact (such as billing and taxes), and compliance with company-specific coding rules. It measures the average of 8 runs based on pass@1 using specific harness combinations such as Claude Code and Codex CLI, rather than standalone model performance.

Resolution Rates and Costs by Model

In evaluations based on actual enterprise production code, Fable 5.1 (Claude Code) ranked first with a 38.8% resolution rate. This was followed by GPT-6 Astra (Codex CLI) at 33.8%, Gemini 3.8 Flash (Gemini CLI) at 31.2%, and GLM 5.3 (Claude Code) at 28.8%. In terms of cost, it was found that an increase in resolution rate does not guarantee an increase in cost, with the estimated cost range being $2.50~$6.96.

• Fable 5.1: 38.8% resolution rate, $6.96 cost
• GPT-6 Astra: 33.8% resolution rate, $4.67 cost
• Gemini 3.8 Flash: 31.2% resolution rate, $2.50 cost
• GLM 5.3: 28.8% resolution rate, $5.12 cost

Task Complexity and Failure Types

With an average of 11 file modifications in reference solutions, the complexity is higher than existing benchmarks (6 files for FrontierCode and DeepSWE). In the 'Analytics stream reducer' task out of 10 tasks, all models recorded a 0.0% resolution rate and failed. Additionally, low resolution rates were observed in multiple tasks such as 'Tax jurisdiction' (3.1%) and 'Linearizable scan' (4.7%).

Analysis of failure types identified Unverified assumption as a primary failure factor. For GPT-6 Astra, 29 out of 67 failures (43.3%) stemmed from unverified assumptions, while Gemini 3.8 Flash had 21 (31.3%) and Fable 5.1 had 6 (9.0%).