What did 29CM's QA team change to dramatically increase test automation usage rate?

29CM's QA team operated test automation by running 28 E2E scenarios based on Jenkins before app deployment. The problem was that the frontend had a frequent deployment structure, and in situations requiring fast verification after deployment, feedback was too slow with the existing approach.

Initially, they separated webview scenarios to run independently and even prepared a dedicated test device. However, usability was low because the development team had to access QA Jenkins directly, and to solve this, they simplified the execution flow with a Slack bot.

The Slack bot ran automation by selecting scenarios and OS via slash commands, and delivered notifications for execution start, progress results, and completion. They actively promoted it too, but after a month of operation, usage remained low, and the cause was clear. This was because the Slack bot also ended up being a structure requiring human intervention.

Afterward, they changed direction and focused on reducing human intervention. They connected it so that a trigger would automatically fire upon frontend deployment, and modified it to distinguish results between the native area and the Webview area, so that only the teams that needed to would receive immediate notifications.

The key changes were as follows.

• If the failure location is Webview, add a frontend team mention.
• Send mentions for each individual scenario case result, rather than after overall completion.
• Even in app automation, if it's a Webview issue, enable the frontend team to recognize it immediately.

They also continued improvements on the operational side. To further shorten the existing execution time of under 20 minutes, they planned to expand test devices from 2 to 3, and newly implemented 10 mobile web browser test scenarios in addition to mobile webview, preparing for parallel execution.

As a result, the existing mobile build-based automation changed to a level of running over 100 times per month on average, and the execution rate rose from under 5% to 100%. The conclusion was clear. Even with clear intent and impact, if the cost of human intervention is high, automation won't be used.