Netflix Study Quantifies Value of Personalized Recommendations via Hybrid Modeling and A/B Tests

Netflix researchers developed a hybrid approach to measure the causal impact of personalized recommendations, addressing the challenge that recommendations are targeted to demand, making it difficult to separate personalization effects from underlying member preferences. The study, published in the International Journal of Industrial Organization and presented at ACM EC '26, introduces a model that treats viewing as a day-by-day process, using a low-rank factorization to capture dynamic member preferences and title similarities.

To validate the model's counterfactual predictions, the team conducted randomized experiments that perturbed recommendation visibility. The model's predicted substitution patterns aligned strongly with experimental ground truth, showing a correlation of 0.86 and an R-squared of 0.73. Using this validated framework, the researchers compared the current personalized system against baselines including random, popularity-based, and matrix factorization recommendations. The results demonstrated that the current system generates substantially higher engagement and consumption diversity than these baselines.

The analysis decomposed the impact of recommendations into exposure, selection, and targeting effects. While mechanical exposure increases consumption probability, it accounts for only about 7% of the gap between the current recommender and a baseline. The majority of the value comes from targeting, which is roughly three times more impactful than exposure. Targeting effects are strongest for mid-popularity titles, where personalization effectively connects specific titles with specific member tastes, whereas exposure effects are more uniform across the catalog.