Raising the Bar on SWE-bench Verified with Claude 3.5 Sonnet

The upgraded Claude 3.5 Sonnet achieved 49% on SWE-bench Verified, surpassing the previous best score of 45%. The key lies not just in the model itself, but in how the agent scaffolding surrounding the model was designed.

SWE-bench evaluates the ability to resolve GitHub issues from real open-source Python repositories. The model is given a checkout of the repository just before the issue and an execution environment, and must understand, modify, and test the code before submitting an answer; evaluation is based on the unit test results from the actual PR.

This piece emphasizes that SWE-bench is not simply a model benchmark, but evaluates the entire model + tools + execution loop. Performance can vary greatly for the same model depending on scaffolding such as prompts, output parsing, and how the next action is reflected.

The agent created by Anthropic for this is intentionally simple.

• It provides only a Prompt, Bash Tool, and Edit Tool, letting the model choose its own problem-solving strategy.
• Rather than enforcing a lengthy procedure, it aims to make maximal use of the model's judgment across exploration, writing reproduction scripts, making fixes, re-running, and checking edge cases.
• Tool design was also treated as important. For example, absolute paths were enforced, and file modifications were designed around string replacement to reduce mistakes.

On the benchmark side, SWE-bench-Verified is a version that narrows the original dataset down to 500 problems reviewed by humans, leaving only tasks that are actually solvable. The piece notes that at the time, no model had yet exceeded 50%, and while this 49% is a very high level, it still sees significant room for further improvement.