Beyond Two-Tower: Redesigning the Serving Stack for Next-Generation Lightweight Ad Ranking Models

Two-Tower, the existing standard for ad recommendation systems, offers excellent computational efficiency but has limitations in modeling complex, high-order features such as user-item interaction features or target attention.

To address this, the team decided to introduce a GPU-based model inference stage capable of directly modeling more complex neural networks. However, the key challenge was solving the latency problem that arises when adding heavy GPU inference to an already optimized serving stack.

The team completely redesigned the serving pipeline to minimize latency. In particular, to reduce the bottleneck in the Feature Fetching stage, they introduced the following Inventory Segmentation Strategy.

• Segment 1 (high-value candidates): For a high-revenue candidate pool of roughly 1M (1 million) items, features were directly embedded as registered buffers within the PyTorch model file. This completely eliminates network I/O and Host-to-GPU data transfer, keeping feature data resident in the GPU's HBM (High-Bandwidth Memory) to maximize processing speed.
• Segment 2 (long-tail candidates): For the remaining candidate pool of roughly 1B (1 billion) items, features are fetched using a combination of a high-performance Key-Value Store and in-host caching.