N-Day-Bench – Can LLMs find real vulnerabilities in real codebases?

N-Day-Bench evaluates whether frontier LLMs can find known security vulnerabilities in real repository code.

Each month it pulls new cases from GitHub Security Advisories, checks out the repository at the last commit before the patch, and provides a sandboxed bash shell. Since static vulnerability detection quickly goes stale and can leak into training data—causing scores to measure memorization—the benchmark manages the contamination window through monthly updates.

Each case proceeds with 3 agents:

• Curator: reads the advisory and writes the answer key
• Finder: the model under evaluation, which explores the code within 24 shell steps and submits a structured report
• Judge: grades the blind submission

The Finder starts without seeing the patch, and must trace actual code paths starting from sink hints.

Target repositories must meet a 10k+ stars threshold, and a diversity pass is applied so that no single repository is overrepresented. Ambiguous advisories—such as merge commits, multi-repository references, or unresolvable refs—are excluded.

Currently evaluating GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, GLM-5.1, and Kimi K2.5, with all traces made public.