LLM Agent Security Patch Benchmark: CVE-Bench

CVE-Bench has been developed to evaluate LLM agents' ability to patch security vulnerabilities, targeting 20 real-world CVEs across 18 Python projects. A total of 300 tests were conducted across 5 frontier models under three prompt conditions (advisory, diagnose, locate).

The key experimental results are as follows:

• Resolution rate: The overall average resolution rate is 50%, rising to 60% when a detailed advisory report is provided. However, test pass/fail is judged strictly based on whether the vulnerability has been completely eliminated.
• Model performance: Performance differences between model families were statistically significant, but differences within the same family did not reach a significant level at this test scale. Notably, higher-cost models did not necessarily guarantee overwhelmingly better performance, suggesting that cost-efficiency can be an important selection criterion.
• Failure patterns: A phenomenon called 'Wrong-search drift' was identified as a major cause of failure, in which a model, after locating the correct file, makes a single incorrect inference and then continues moving in the wrong direction based on that error for the remaining steps.