Why Doesn't Deployment Work? Designing ML Model Training and Evaluation to Match Target Metrics

In production ML, metrics aligned with business goals must be determined before accuracy. Optimizing for the wrong metric can improve offline performance while having almost no real-world impact after deployment.

Hyperconnect AI worked on the problem of selecting the primary attribute among an item's various attributes—the one that would generate the highest conversion rate. They first approached this with supervised learning, but found its limitations due to the gap between recommendation logic and user distribution. They then progressively narrowed the problem to user-wise CR ranking, which compares converted/non-converted attributes within the same user, and item-wise CR ranking, which compares attributes within the same item. However, the training objective could still be skewed by the recommendation system and hidden confounders.

The key to the solution was randomized attribute shuffle data. By randomly assigning the primary attribute among an item's attribute candidates during data collection, they cut off the influence of confounders, and ultimately adopted item-wise attribute CR ranking as the training objective. The model learns to score the attribute with higher conversion rate within the same item via score differences, optimized with a pairwise dataset and ranking loss.

For evaluation, they noted that simple mean Spearman's rank correlation coefficient (SRCC) or top-1 accuracy don't properly reflect real business impact. This is because the observed conversion rate itself is a noisy estimate, and the scale of conversion rate differences varies by item, distorting meaning.

Instead, they defined relative mean conversion-rate lift for offline evaluation. By calculating the average conversion rate increase of model-selected attributes versus human-selected attributes on held-out attribute shuffle data, they could directly gauge the expected impact upon deployment. They also reduced maximization bias by splitting data into two groups, similar to the Double DQN approach, and estimated a lower bound for optimal performance.

The model trained with this approach achieved the best offline results, and subsequently, in a large-scale online A/B test, it confirmed significant mean CR lift and improvements in key metrics, successfully making it to actual deployment.