Closing the Gap: Diagnosing the Online–Offline Discrepancy in Pinterest's L1 Conversion Model

In Pinterest's L1 ranking stage, a new conversion(CVR) model consistently won offline, but in online A/B tests, CPA repeatedly came out neutral or worse. The authors defined this as an Online–Offline(O/O) discrepancy and traced the causes by examining the model, serving, and funnel together, rather than assuming a single bug.

First, the offline evaluation itself was not the problem. The team recomputed LogMAE and calibration across three log sources (auction winners, full-request auction candidates, partial-request auction candidates), and also compared results by pCVR percentile. The experimental model showed substantially lower log-loss than the production model across all datasets, and performance was maintained or improved within each bucket. In other words, the offline improvement was real, and evaluation bugs alone could not explain the online underperformance.

Exposure bias and traffic share were not the core cause either. A ramp experiment raised the treatment share from roughly 20% to 70%, but even with more traffic going to treatment, the over-calibration issue persisted. In addition, timeout and p50/p90/p99 latency for the query tower and Pin tower were not significantly worse than control, so serving failure was also not the main explanation.

The real problem was a mismatch in features and embeddings.

• Feature O/O discrepancy: Training and offline evaluation used rich logged features, but the online L1 embedding builder didn't include some high-value Pin features at all.
• Examples of missing features: targeting spec flags, offsite conversion visit counts (1/7/30/90 days), annotations, and MediaSage image embeddings.
• As a result, the model learned during training to rely on these signals, but at serving time it only operated with a thinner feature set.

To fix this, they modified the UFR config to onboard the missing features into the L1 embedding path. As online feature coverage was restored, online loss for the CVR and engagement models improved as well. At the same time, they changed the default behavior of the UFR tool so that features onboarded to L2 are automatically considered as candidates for L1 embedding use, reducing silent O/O issues.

The second problem was embedding version skew. In a two-tower architecture, the query tower and Pin tower may not use the same checkpoint—and online, due to hourly indexing snapshots, separate rollouts, and long index build/deploy cycles, embeddings of different versions end up coexisting within a single index. Experiments varying the Pin embedding version while holding the query tower fixed showed that simple models were affected only in a limited way, but complex models like DHEN saw loss noticeably worsen in some slices as skew increased.

Afterward, rather than trying to eliminate this entirely, the team accepted it as an operational constraint. For large tiers, batch embedding inference is used to produce a single-version embedding, and new model families are now required to pass a version-skew sensitivity check.

Finally, they also confirmed that even good predictions may not translate into CPA improvements across the full system. Retrieval, L1 ranking, L2 ranking, and auction each have different objectives and constraints, and improving L1 metrics alone doesn't help end-to-end performance unless retrieval recall or ranking recall move together with it. In practice, on some surfaces, offline L1 metrics improved but recall was already near ceiling, so there was no online effect—whereas treatments where recall actually moved were the ones that produced clear online wins.

They also clarified that offline LogMAE, KL, and calibration live in an entirely different world from online CPA. Offline evaluation often treats L2 predictions as teacher labels, but online CPA is affected by bid, budget, pacing, and auction logic. Even if candidates are supplied better in terms of downstream utility, the expected CPA improvement can disappear once it passes through actual auction behavior—meaning offline metrics are a necessary but not sufficient condition.

In the end, the conclusion of this work is clear: the model, embeddings, and feature pipelines must be treated as a single system, the funnel determines the ceiling, and debuggability itself is part of the model product. Only by building tools like coverage dashboards, embedding skew tests, and parity harnesses into the launch process by default can future L1 experiments be less shaky between offline and online, and actually translate into real business outcomes.