Toss QA Redesigns Hotfix Criteria for Gradual Rollouts and Introduces AI-Based Crash Analysis

The Toss QA Platform team discovered that existing severity criteria (Critical, Major, etc.) were unsuitable for hotfix decisions in gradual rollout environments and redesigned them into a structure that determines whether to proceed or defer to the next deployment. This is because issues affecting only 1% of users must be considered for potential risks, while conversely, issues occurring only under specific conditions can lead to new risks if forced hotfixes are applied.

Hotfix Decision and Verification Process

The decision to proceed with a hotfix is made by consensus between the Release Master and the QA Master, considering deployment and quality perspectives. Immediate action is taken for issues such as core functionality paralysis, revenue impact, or regulatory violations, while issues with low usage rates or clear workarounds are deferred to the next regular deployment. Time-based buffers were removed to focus on 'sufficient review'.

To reduce reproduction time during crashes, a crash-path tool based on Claude skills was developed. This tool automatically collects data from Bugsnag (Android) and Firebase Crashlytics (iOS), analyzing user behavior logs and environment information. Rules are applied to prevent the AI from arbitrarily guessing screen names or reproduction signals, ensuring consistent results.

Record Management and Recurrence Prevention System

Hotfix records were integrated into the Tossion platform to resolve format differences between iOS and Android and make them accessible to everyone. Hotfixes that were not proceeded with are also recorded with reasons to serve as a basis for future decisions. Initially, post-hoc logging was made cumbersome to reduce hotfixes, but this led to poor record-keeping and failed. Consequently, the approach was shifted to having an AI bot automatically generate drafts of post-hoc records based on channel conversation context, lowering the burden while preserving context.

Every month, the Android, iOS, and QA teams gather to review hotfixes and check recurrence prevention actions, operating a closed loop that verifies actual implementation rather than just commitments.