Pinterest Enables Multi-Epoch Embedding Training with 1.09% Online Save Lift

• Pinterest enabled multi-epoch training for its Homefeed ranking model, achieving a +1.09% lift in online saves by decoupling embedding table optimization from the rest of the network.
• Freezing the embedding table during early training epochs and allowing adaptation only in the final epoch prevented catastrophic overfitting, outperforming both single-epoch baselines and fully frozen multi-epoch configurations.
• A semi-synchronous training pipeline overlapped distributed embedding communication with computation, reducing GPU idle time caused by sparse ID lookups that previously consumed over 15% of total training time.
• System optimizations including fused ID lookups and specialized singleton-row operations reduced collective latency and kernel count, while gradient accumulation amortized optimizer communication costs.

Training Strategy

• Delayed embedding updates: Keeping the table frozen for the first epoch and co-training in the second yielded a 1.16% lift on a 20-day data window, compared to a 0.80% regression when co-training started from the first epoch.
• Gradient isolation: Blocking gradients from head and pairwise losses while retaining the contrastive path prevented overfitting but underperformed the fully frozen strategy.
• VLM optimization: The delayed scheme also allowed newly added Vision Language Models to optimize longer without causing embedding table degradation.

System Efficiency

• Semi-synchronous pipeline: A custom implementation using a lookahead dataloader and prefetching forward wrapper overlapped embedding lookups for the next batch with computation for the current batch.
• Bounded staleness: Prefetched embeddings had a delay of one update, which showed no model-quality degradation due to low overlap in sparse IDs between consecutive batches.
• Kernel optimizations: Fused training-ID lookups reduced collective calls by up to a factor of three, and replacing atomic scatter-add with dense GEMM operations improved gradient computation efficiency for small embedding tables.