AI BriefingCollinear AI releases CWE-bench v2 with 100 public security tasks
·2026.11.03 09:00
Key point
The benchmark covers 91 CWEs with deterministic scoring, where the top model still fails 19% of tasks.
Details
- Collinear AI released CWE-bench v2, a public set of 100 tasks testing agents on finding and fixing security weaknesses in real open-source repositories.
- Gemini 4 Argon leads the leaderboard with 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% pass@1.
- 95 of the 100 tasks are solvable by at least one model, but the top performer fails 19% of the set on every attempt.
- DeepSeek V4.1 Flash offers the lowest cost at $0.04 per rollout for 57.0% pass@1, while Gemini 4 Argon costs $3.92 per rollout.
- The benchmark uses Harbor for execution, allowing direct runs from the Hugging Face Hub with deterministic, programmatic verifiers.
- Tasks are tagged as RL environments, enabling the use of the same verifiers for training defensive agents.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.