AI BriefingAI Security Patch Success Rate Reaches 25%
Key point
Research by Off-by-1 Labs indicates that the automatic patching success rate for security vulnerabilities by AI models is approximately 25%, rendering them unreliable.
Details
An analysis by security research firm Off-by-1 Labs on the ability of the latest AI models to patch code security vulnerabilities revealed that the success rate is only about 25%.
The researchers identified the following limitations in AI's ability to resolve security issues:
- Low success rate: The proportion of non-trivial security issues fully resolved was one out of four.
- Creation of new vulnerabilities: During the patching process, AI often failed to fix existing problems or instead introduced new security vulnerabilities.
- Excessive review costs: Properly verifying patches proposed by AI requires high cognitive effort from humans, which is practically inefficient.
In conclusion, the research team judges that current AI models are unreliable for autonomously fixing security vulnerabilities, and that manual patching by humans is safer and more efficient.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
DeepSeek V4.1 Flash Successfully Exploits All 11 Vulnerabilities in AI Hacking Benchmark
Rootless Linux Sandbox 'Drop' Released for Isolating AI Agents and Third-Party Programs
Fake Customer Service Pages Induce ChatGPT and Gemini to Display Scam Numbers
Taste-Bench: Frontier LLMs Achieve 59.7% Accuracy on Decision Forking TasksNSA Spending Billions to Test Frontier AI Models for Security Vulnerabilities