Yeogi-eottae Processes 280 Overseas Tour QA Tickets with AI Agent, Achieving 56% Same-Day Resolution Rate

The Yeogi-eottae Experience Development team shared a case study where they used an AI agent to process a total of 280 QA tickets during the launch of their overseas tour service. Even with a single frontend developer covering an expanded scope, the same-day resolution rate for this project rose to 56%, compared to 23–48% in past projects (such as package tours), resulting in shortened lead times.

QA Pipeline and Role Division

The process was structured as a five-stage pipeline: ticket intake, reproduction, root cause tracking, review, and deployment. The core rule was "no code changes before confirming the root cause." Humans judged the cause and direction, while the agent reduced time spent on verification tasks such as reproduction, investigation, and documentation.

Simulator Usage and Debugging

Issues such as scroll problems on iOS 16 and below, white screens when pressing back on Android, and authentication issues in older app versions were reproduced and diagnosed using simulators via tools like xcrun simctl, Playwright WebKit, and adb. By having the agent directly launch simulators to compare differences across versions, processing time decreased significantly.

Knowledge Sharing via MEMORY.md

To compensate for the agent's lack of memory between sessions, a MEMORY.md file was introduced. API documentation locations, discrepancies between actual responses and documentation, and project status were recorded in Markdown files, which the agent referenced at the start of each session to improve efficiency in repetitive tasks and context maintenance.

Failure Cases and Lessons Learned

Failures occurred due to Git staging area conflicts, arbitrary removal of verified code, and implementations based on incorrect assumptions. These experiences led to the establishment of rules such as "confirm direction before writing code" and "protect verified code." The author concluded that teaching the agent its scope of responsibility and prior knowledge is more important than ticket processing speed, and that a step-by-step training approach was more effective than providing standardized answers.