Spotify: Bayesian A/B Testing Is Not a Single Method… Default Settings Are Numerically Equivalent to Frequentist Approaches

Spotify emphasizes that Bayesian inference is a collection of various configurations defined by stopping rules, priors, and likelihoods. The flat prior and stopping based on posterior probability thresholds, which are used as defaults on most platforms, reproduce the same false positive rates as frequentist peeking. Additionally, with a flat prior and a two-group normal model, Bayesian and frequentist procedures yield numerically identical results. The potential advantages of Bayesian methods, such as shrinkage via Empirical Bayes priors and false discovery rate control, require well-calibrated historical priors, and performance can degrade if the prior is misspecified. Therefore, adopting Bayesian methods is not merely a change in statistical technique but a complex decision involving prior management and maintenance costs, suggesting that default settings alone may offer no substantive superiority over frequentist approaches.