Meta Uses Idle AI Inference Capacity to Improve Ads Recommendations

Meta has developed a system that aggregates locally idle inference capacity across its fleet to improve ads recommendations. This approach addresses the inference capacity paradox, where a fleet is globally constrained but contains significant pockets of unused resources due to mismatches in demand, hardware types, and regional availability. By utilizing this opportunistic capacity, the system increases ad relevance and advertiser outcomes without additional capacity orders.

Two Elasticity Levers

The system employs two complementary mechanisms to safely consume volatile capacity:

• Rank More Ads (RMA): Increases the number of ad candidates evaluated by ranking stages when both CPU and accelerator capacity are available.
• Serve Better Models (SBM): Routes a higher share of requests to more powerful models when accelerator-only capacity is surplus, keeping upstream CPU demand stable.

Safety and Isolation

To protect baseline performance, the system uses physical isolation for SBM and logical isolation for RMA. Physical isolation ensures baseline and powerful models use separate routing targets, while logical isolation limits RMA work to a global budget supported by opportunistic replicas. If capacity is reclaimed, the system degrades gracefully by reducing optional work rather than overloading dedicated baseline capacity.

Control and Load Shedding

The system relies on feedback loops to adjust workload dynamically. RMA uses a PID controller to tune candidate volume based on tail latency, while SBM uses a central controller to manage model coverage. For sudden capacity loss, value-aware load shedding at both server and client levels reacts within milliseconds to drop lower-value work, ensuring latency and reliability targets are met.