Spotify Analysis: Offline Evals Can Replace A/B Tests Only Under Strong, Hard-to-Verify Assumptions

·2026.11.19 22:34

Key point

Spotify engineers demonstrate that naive estimates of the offline-online correlation are attenuated by measurement error, showing 24% when the true value is 60%.

1 / 11

Details

  • Offline evals can replace A/B tests for shipping decisions only if the conditional distribution of online effects given offline results is known, a condition requiring strong, hard-to-verify assumptions.
  • Measurement error significantly attenuates the observed correlation between offline and online metrics; in Spotify's working example, a naive estimator reports 24% squared correlation while the true value is 60%.
  • A decision gate can be constructed to bound the risk of harmful shipping decisions, but thresholds must be adjusted based on the strength of the surrogate relationship; weak correlations require stronger evidence from offline evals.
  • Pooling heterogeneous experiments can create Simpson's paradox, where a strong pooled relationship masks the absence of a relationship within individual change classes.
  • LLM-as-a-judge bias that depends on the treatment arm can flip the sign of the observed effect, making worse versions appear better, whereas arm-independent bias cancels out in treatment effect estimation.
  • Deploying the gate stops A/B testing for shipped changes, causing a loss of ground truth data in those regions; a random audit pool is required to detect drift in the offline-online relationship over time.
  • The approach is viable in principle but faces high practical barriers: maintaining a homogeneous dataset, handling distributional assumptions, and characterizing LLM judge biases remain difficult and require continuous reevaluation.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.