GEM Training: How Meta Doubled the Efficiency of an LLM-Scale Ads Foundation Model

Meta's Generative Ads Recommendation Model (GEM) is the foundation model behind ad recommendations on Instagram and Facebook. GEM is trained at LLM scale on thousands of the latest GPUs, and over the past 12 months, Meta increased total training compute by 4x while doubling end-to-end training efficiency, achieving 20-25% Model FLOPs Utilization (MFU).

GEM is a hybrid architecture combining trillions of sparse embedding parameters with billions of dense parameters. Built on ad content and user engagement data, it processes sequential features such as user activity history and non-sequential features such as position and ad creative representations separately, while also learning the interactions between features.

The data and compute characteristics of recommendation models make it difficult to directly apply typical LLM training infrastructure.

• User activity histories vary greatly in length, and using padding can waste up to 50% of compute.
• Asymmetric tensor shapes—such as long sequences with short attention windows, or long queries with short keys/values—make it hard to saturate GPU compute units.
• MLPs with small embedding dimensions and various normalization operations are memory-bandwidth bound, lowering GPU utilization.
• CTR/CVR prediction is sensitive to numerical changes, making simple low-precision training difficult to apply.

Meta improved both compute efficiency and scaling efficiency. On the compute side, Meta developed kernel libraries tailored to recommendation workloads, including Jagged Flash Attention (JFA), Generalized Dot-Product Attention (GDPA), and BlockAttention, and optimized mixed ultra-low-precision training—including MXFP8-based attention and MLPs—for the latest GPU architectures.

On the scaling side, Meta applied network-topology-aware 5D parallelism. It combined 2D FSDP and Expert Parallelism for dense parameters with Fully Sharded 2D Model Parallelism for sparse parameters, and co-designed Streaming Multiprocessor (SM)-free collective communication with Meta's multi-tier network architecture to reduce communication overhead.