Banksalad Unveils Experiment Analysis Infrastructure V2… Ensuring Reliability and Scalability with an Automated Pipeline

Banksalad's Experiment Platform team built an experiment analysis infrastructure to quantitatively verify the impact of product improvements. In the initial V1 stage, manual analysis was performed based on Python Notebooks, but as the number of experiments increased, problems such as human error, maintenance difficulties, and resource inefficiency arose.

To solve this, the V2 automated experiment analysis pipeline was developed, consisting of three core components: Metric Configuration, Data Preparation, and Metric Calculation. This pipeline automatically collects, calculates, and loads experiment result data, and verifies statistical significance using a Two-sample z-test.

The Metric Calculation stage was separated into four steps—Experiment Event, Subject-level Aggregated, Metric-level Aggregated, and Experiment Analysis Result—considering reusability and flexibility. Each step proceeds independently to improve analysis efficiency, and the final results are loaded into AWS S3 and then used for Rollout or Rollback decisions through the experiment platform UI.

With the introduction of the automated infrastructure, the burden of repetitive work on Data Scientists has been reduced, and any team member can now easily check reliable experiment results through the UI. This has contributed to maturing a data-driven decision-making culture, and further enhancements are planned, including strengthening statistical power and supporting more complex metrics.