Invisible Quality, Data: If the Logs Are Wrong, So Are the Customers

Mobile apps have fixed release cycles, so if log issues are discovered after release, fixes come late and the impact lingers. Incorrect events accumulate in operational logs and can even shake up recommendations, rankings, and experiment results, ultimately affecting the customer experience directly.

To prevent this, log review was added to the final R/T (Regression Test) stage. Appium-based UI tests automate actual user actions, record timestamps before and after specific actions, and then validate only the logs generated within that time window.

Log collection leveraged GA4 and the internal event collection system Heathrow, respectively. GA4 filters BigQuery query results based on time, while Heathrow filters Charles packet parsing results based on time to find events.

The review works by identifying events using event_name, landing_url, and path, then checking the parameters in the payload. Common parameters are managed as a Spec, so only case-specific conditions need to be declared to reuse the same validation.

For example, ga_session_id, device_os, device_type, and channel_type are checked as common required values, and for Heathrow, entry into a brand shop is determined by looking at both click_content and whether /brand/ is included. Validation results—success or failure—are automatically notified via Slack and saved as a report.

The biggest benefits were standardizing validation quality and reducing QA resources. By automatically checking events and conditions that were easy to miss with manual review, issues such as missing non-member user attributes on Android, incorrect device_os collection on iOS, and missing brand shop logs for a specific brand were caught before release.

During the automation process, it was also discovered that on iOS, force-terminating the app with driver.terminate_app() caused GA logs to be lost because they weren't flushed. To resolve this, the scenario was changed so that instead of terminating immediately at the end of the test, the app is switched to the background before termination.

The architecture was later transitioned to Snowplow → Databricks → S3, but the core principle remained the same. It confirmed that automatically collecting logs and validating them against defined criteria at the final stage before app deployment is itself a QE strategy that reduces release risk.