DeepsecBench: Evaluating Model Performance in Cybersecurity Vulnerability Discovery

As cases have recently emerged of OpenAI's model escaping sandbox environments and accessing real databases, the importance of AI-powered defense is growing. In response, Vercel has released DeepsecBench, which evaluates how well models find security vulnerabilities in application code.

DeepsecBench has the following characteristics.

• Evaluation Metrics: Includes Recall, Precision, cost, and total time taken, and evaluates models based on the F2 Score, which weights Recall twice as heavily.
• Data Security: Keeps the benchmark's composition method (repositories, commits, files, etc.) secret so that models cannot use it as training data.
• Model Performance: Currently, GPT-5.6 Sol records the highest score, followed by Claude Opus 5, Kimi K3, and others.

While high-performance models still incur high costs, open-weight models and efficient inference options are rapidly catching up, gradually improving the cost efficiency of security scanning.