Automating Service Vulnerability Analysis with LLMs #2

To efficiently deliver large-scale source code, we built SourceCode Browse MCP, pre-indexing symbols with ctags and then parsing function scopes with tree-sitter. The tool was composed of 4 functions—find_references(), read_definition(), read_source(), get_project_structure()—so that the AI could accurately find definitions and references like an IDE.

To improve analysis consistency, we turned SAST from a vulnerability detection tool into a device for extracting all candidate input paths. Using semgrep's taint rule, we extracted both untrusted input sources and sinks, received them as SARIF, trimmed them down to only the necessary information into JSONL, and merged consecutive lines to reduce token waste.

The cost and efficiency problem was solved with a Multi-Agent structure. We changed the approach so that a Discovery agent, looking only at the given snippet, selects paths with a high likelihood of vulnerability, and an Analysis agent verifies those paths. To avoid filtering too strictly here, the System Prompt explicitly stated the trade-off: "avoid using MCP as much as possible" and "include it if there's even a slight possibility." As a result, we succeeded in selecting 144 out of 286 paths and finding 27 vulnerabilities.

The last wall was sustainability. Since Cloud Models alone could not handle the cost of daily analysis across hundreds of services, we compared Open Model candidates Qwen3:30B, gpt-oss:20B, and llama3.1:8B. Using a sample web project containing XSS, IDOR, Deserialize, and Path Traversal, we measured true positive rate, false positive rate, and analysis rate, and also evaluated MCP Tool calling stability and token efficiency together, ultimately selecting Qwen3:30B.

The unstable response format of the Open Model was compensated for with Pydantic and Instructor. We enforced the output schema through type hints and validation, and induced automatic correction of invalid responses to reduce non-determinism. In the end, the key was not a single smarter inference, but combining accurate code exploration, candidate pool reduction, agent division of labor, and model validation to build an analysis system that could keep running in practice.