Real-SWE Benchmark Reveals Limits of AI Coding Agents in Handling Ambiguous Requirements

The Real-SWE benchmark evaluated the performance of AI coding agents in real enterprise codebase environments. The tasks used in the evaluation included complex business logic such as Multi-region sweep, Tax jurisdiction, and Analytics stream reducer, where models faced situations requiring them to infer unspecified requirements.

Failure analysis revealed that the most frequent causes of failure were 'Unverified assumption' and 'Missed requirement'. Looking at the Task Resolution Rate, Multi-region sweep showed a relatively high resolution rate of 67.2%, but resolution rates plummeted for complex tasks such as Tax jurisdiction (3.1%) and Analytics stream reducer (0.0%). In particular, some tasks saw all models fail or only a very small number succeed.

Comparing performance by model, GPT-6 Astra and Gemini 3.8 Flash recorded a resolution rate of 8/8 in Multi-region sweep, demonstrating high performance, while GLM 5.3 and Kimi K3 recorded a low resolution rate of 2/8 on the same task. Grok 4.6 recorded high token usage of 261k and 315k in Linearizable scan and Analytics stream reducer, respectively, but failed to resolve them or showed low performance. This suggests a limitation where models consume excessive tokens during complex reasoning processes yet fail to accurately handle ambiguous requirements.