Vulnerability Research: Small-Model Repetition Wins Out

Hacktron targeted oauth2-proxy v7.15.0, using two independently discovered 0-day auth bypasses as ground truth to compare recall and cost across models. After finishing preprocessing with call graph, documentation, configuration, and code enrichment, only the LLM stage was varied, and since validation hasn't happened yet, this comparison looks at recall rather than precision.

• The ground-truth vulnerabilities were CVE-2026-34457 (auth bypass via health check User-Agent matching in auth_request mode) and CVE-2026-40575 (bypass via X-Forwarded-Uri header spoofing).
• Larger models like Claude Opus 4.6 and Gemini 3.1 Pro found both vulnerabilities more reliably, but still didn't reach 100% reproduction.
• GPT-5.4 Nano caught only about 10% of Finding A, but with multiple repetitions this rose to nearly 90%.
• On a cost basis, instead of 1 run of Gemini 3.1 Pro you could run Flash 14 times, and instead of 1 run of Claude Opus 4.6 you could run GPT-5.4 Nano 28 times, making the repetition strategy more favorable.
• As an exception, the Gemini 3.1 Flash line produced the strongest results, at 9/10 and 10/10 levels.
• The conclusion is that while frontier models are strong for security analysis involving an experienced researcher in the loop, in automated environments without a human, running smaller models multiple times is better in terms of cost-to-recall.