New Code Benchmark DeepSWE Released

DeepSWE is a new code-writing ability evaluation metric designed to overcome the limitations of existing benchmarks. Its main features are as follows:

• Contamination free: Rather than transforming existing commits or PRs, it uses tasks written from scratch, blocking the possibility that models encountered the answers beforehand during training.
• High diversity: It provides a wide range of tasks using 91 different repositories across 5 languages.
• Realistic complexity: While the prompt length is about half that of SWE-bench Pro, the amount of code required to solve tasks is 5.5x greater and the number of output tokens is 2x greater, making it more similar to real engineering environments.
• Reliable verification: It uses manually written verifiers that test the actual behavior of the software rather than implementation details.

The project has been released as open source, so anyone can use it.