Automating Service Vulnerability Analysis with LLMs #1

Toss disclosed the four key challenges it faced while using an LLM to automate service vulnerability analysis, along with their solutions.

The first challenge, handling large-scale source code, was solved by introducing MCP (Model Context Protocol). It developed SourceCode Browse MCP, which allows the LLM to directly explore the source code whenever needed, overcoming token limit issues and hallucination phenomena.

The second challenge, analysis accuracy and consistency, was solved by combining SAST tools such as Semgrep. By first collecting all Source $\to$ Sink paths and then having the LLM analyze them, it established a strategy to accurately find vulnerabilities without missing any.

The third challenge, cost, was optimized with a Multi-Agent system. The Discovery agent first filters out unnecessary paths, and the Analysis agent is designed to process only paths with actual potential vulnerabilities, dramatically reducing token consumption.

Finally, for sustainable operation, it switched from a Cloud Model to an Open Model. After comparative testing of multiple models, it ultimately selected Qwen3:30B, which showed excellent analysis rate and true positive rate, achieving a high accuracy of over 95%.