Why KakaoPay developed its own AB testing service

When companies launch new services, accurately predicting user reactions is difficult. AB testing is a core process that observes user behavior to identify Wow Points and Pain Points, reducing costs through data-driven decision-making.

KakaoPay used the open-source Wasabi for initial experiments, but there were limitations to expanding it company-wide. It was difficult to integrate with company data, and it was impossible to run customized experiments that set experiment targets based on detailed conditions such as specific devices or a user's purchase history.

To solve this, KakaoPay developed its own service reflecting the following key requirements.

• Flexible user segmentation: Using ANTLR, the service was designed to precisely target experiment subjects based on various conditions such as device information, app version, and user demographics.
• Failure response system: To provide stable service even during server failures, a Fallback bucket was introduced so that users can see the existing stable screen when a problem occurs.
• Statistical analysis: The company aimed to build an environment that can select high-performing Winner Buckets based on experiment data.