Detecting High-Severity Security Vulnerabilities Using Open Models

Ramp used its internal coding agent, Inspect, to deploy about 10,000 agents against its own backend over 8 hours. This uncovered 7 high-severity security vulnerabilities and numerous low-severity vulnerabilities that previous penetration testing, bug bounties, and AI scans had failed to find.

This scan was conducted in a model-agnostic way that did not rely on any specific closed model. It demonstrated that cheap open-weight models alone are sufficient to identify high-severity security issues.

The key to scaling is compute utilization. The principle that increasing the number of LLM attempts on programming tasks improves success rates log-linearly was applied to security work. This approach is highly effective for security scanning because what matters is not getting the right answer on a single try, but finding just one valid attack path among many attempts.

The overall process consisted of the following:

• Massively parallel deployment: Ran Inspect sessions against every endpoint, focusing intensively on security-sensitive areas such as authentication and payments.
• Deduplication: Of the 6,000 initial tickets, duplicates were merged using Inspect, compressing them down to about 3,000 unique tickets.
• Triage: Finally, real bugs were distinguished from false positives, ranked by severity, and converted into actionable tasks.