Toss Doctor V2 Released: Maximizing QA Efficiency with LLM-based Automated Testing and Self-healing

The Toss QA Platform team released Toss Doctor V2, an automated smoke testing tool. While V1 reduced the burden of repetitive manual testing, it had limitations such as tests breaking easily upon UI changes and requiring significant manpower for root cause analysis. V2 significantly improves this by integrating the processes of code generation, element exploration, failure judgment, and recovery with LLMs.

LLM-based Code Generation and Efficient Element Exploration

The test writing approach shifted from procedural code to scenario-based Gherkin syntax. Humans only write the verification content, and the /codegen command utilizes LLMs and Appium MCP to analyze actual device screens and generate executable code. For element exploration, a three-step strategy was applied that first checks screenshots and a list of visible elements instead of passing the entire UI tree, reducing token costs.

Smart Finder and Self-healing Features

If a test fails during execution due to unexpected pop-ups or UI changes, the Smart Finder intervenes immediately. It analyzes the failed screen to remove obstructing elements or find alternative elements to continue the test. In the verification (then) stage, similar matching is prohibited to prevent false positives, but upon failure detection, /diagnosis analyzes the cause and determines whether it is recoverable. If deemed recoverable, executing /recovery (such as modifying locators) resolves the issue without human intervention and logs it.

Expansion to Toss Checker

The technical foundation of Toss Doctor V2 has been expanded into Toss Checker for broader verification. Toss Checker automates not just smoke tests but also regression tests, automating tasks that previously required three testers for two days into 1,375 test cases. This allows QA personnel to move away from simple repetitive tasks and focus on verification design and strategy formulation.