Relentlessly Reducing Failure Rates in Android UI Automation

The 29CM QA team operated Android UI automation for BVT (Build Verification Test) purposes before app releases, but in the second half of 2023, numerous failures occurred even after scenarios were completed, driving up maintenance costs.

The first issue they encountered was that during the build process, missing ids or path changes would break existing element locators. In these cases, they would immediately reflect the changed path, or if an id was missing, check with the development team, trigger a rebuild, and rerun the automation.

A more persistent problem was the window handler not being correctly captured during WebView transitions. They initially assumed that only the last handler needed to be used when moving from native to WebView, but in reality, multiple handlers stacked up in a nested state, and their position varied by scenario, so simple rules couldn't control it.

To reduce this problem, they changed their approach several times.

• Tried using the second-to-last handler instead of the last handler
• Tried specifying a fixed handler position for each scenario
• Tried separating work by restarting the app after certain scenarios to reset the handler history

Through this process, the failure rate went down, but the problem remained that if a scenario failed midway, handler control became impossible again, and a side effect emerged where Slack notifications were split into up to 6 separate messages. Ultimately, they carried out a structural refactoring: making scenarios more independent, restarting the app on completion to clear the history, and consolidating the notification structure back into one.

Another axis of the problem was the difference between the CI/CD pipeline and local execution. Scenarios that ran fine locally via STF led to element search failures or app termination in the pipeline, with issues occurring particularly often on the very first scenario and right after screen transitions.

The cause was a speed mismatch where the driver attempted element search before the screen finished loading. To reduce this, they added brief wait times between actions to control execution speed, and established a verification procedure across three stages—local execution, STF connection, and an actual connected PC—to re-verify problem points.

Through this iterative process of fixes and verification, they significantly lowered the fail rate compared to the initial state, and they continue to track maintenance, scenario enhancement, and execution time reduction.