How Amazon Achieves Global-Scale Vulnerability Detection with Agentic AI

As NVD disclosed more than 48,000 new CVEs in 2025, reducing the time between vulnerability disclosure and defense became a core challenge for security teams. To address this, Amazon built an agentic AI system called RuleForge, which generates and validates detection rules based on publicly disclosed exploit proof-of-concept code.

The existing workflow required analysts to handle every step manually: downloading and analyzing exploits, writing detection logic, checking accuracy through log queries, iterating until false positives were reduced, and finally deploying only after a fellow security engineer's code review.

RuleForge breaks this flow into a multi-agent pipeline that mirrors the human expert's workflow.

• Automated collection and prioritization: Downloads public exploit PoCs and prioritizes them using content analysis and threat intelligence.
• Parallel rule generation: A generation agent on AWS Fargate and Amazon Bedrock simultaneously creates multiple candidate detection rules for a single CVE.
• AI evaluation: A separate judge model evaluates each rule based on sensitivity and specificity. The key is that the generation model does not evaluate itself.
• Multi-stage validation: Synthetic tests confirm basic detection capability, traffic logs such as MadPot verify real-world behavior, and failed rules are sent back to the generation agent with specific feedback.
• Final human review: The best rules must still go through a security engineer's code review, as before, before entering production.

The separate judge model proved especially effective. When the generation model was asked to state its own confidence, it judged nearly all results as good, but introducing a separate judge reduced false positives by 67% while maintaining the number of true positives. Negative phrasing — reframing the evaluation question from "does it match well" to "where does it fail" — along with domain-specific prompts reflecting the criteria security engineers actually use, significantly improved calibration.

The system also generates reasoning chains and compares them against human judgment. In the evaluation, the AI judge's reasoning matched human experts' judgment in 6 out of 9 rules. For example, one person judged a SQL injection regex to be too loose, and the judge similarly concluded that the pattern matched a much broader set of strings than the specific vulnerability warranted.

Amazon deployed this confidence scoring system in August 2025, and over the last four months of that year, RuleForge increased the speed of detection rule generation and validation by 336% compared to manual work. The key is not removing humans from rule writing, but restructuring the process so that generation and evaluation are separated, multiple specialized agents produce drafts, and humans give final approval. This approach is presented as a practical pattern for updating defensive rules faster in large-scale vulnerability disclosure environments.