DeepSWE: A Coding Agent Benchmark for Long-Horizon Engineering Tasks

DeepSWE is a long-horizon software engineering benchmark for measuring the performance of coding agents. To overcome the limitations of existing benchmarks, it achieves four key advancements.

• Contamination-free: Instead of modifying existing commits or PRs, it uses tasks written entirely from scratch, blocking the possibility that models may have seen the answers during pretraining.
• High diversity: It provides a broad pool spanning 5 languages and 91 repositories.
• Real-world complexity: While prompt length is short, the amount of code required to solve each task is 5.5x greater than existing benchmarks, and output tokens are about 2x more, resembling actual development environments.
• Reliable verification: It uses manually written verifiers that test the behavior of the software rather than implementation details.

The existing leading benchmark, SWE-bench Pro, has limitations such as an average of only 120 lines of code per task and a high error rate in the verification process. In contrast, DeepSWE more clearly distinguishes performance differences between models, reflecting the differences in agent performance that real developers actually experience.