Experimenting Better with LLM Evals: The Funnel Is Not a Fork in the Road

Spotify views LLM evals not as a replacement for A/B testing but as an upstream validation funnel. According to the article, only about 12% of Spotify's A/B tests lead to a positive launch, but about 64% yield valid learning by catching regressions or refining hypotheses. Schultzberg and Ottens (2024) call this structure the evaluation funnel.

• Evals quickly validate quality axes that are hard for humans to check at scale, such as relevance, coherence, tone, and intent alignment.
• Experiments confirm whether real users and systems respond as predicted, while also watching guardrail metrics like reduced session length, increased crash rate, and worsened retention.
• Filtering candidates with evals first raises the experiment hit rate, and subsequent experiments then validate the actual effect.

LLM eval is also at the same time a hypothesis generation tool. Once a recommendation pattern that damages trust is found and fixed, the same LLM judge can be used to re-check whether it improved, but an experiment is needed to confirm whether that change leads to long-term engagement or churn prevention.

The issue is that eval is also a proxy metric. Even if the judge prefers option A, it may diverge from the long-term user experience, and conversely, long tasks or long-term behavior changes may not be well captured by offline scores.

What's needed, then, is a verify-validate-calibrate loop. First validate with eval, then confirm user impact with an A/B test, and then run eval again on the test data to narrow the offline-online gap. This is also why Spotify rolls back about 42% of launch experiments, to prevent deterioration in secondary metrics like session length, crash rate, and retention.