Netflix's MediaFM: A Multimodal AI Foundation Model for Media Understanding

As mechanically understanding media became increasingly important for Netflix's vast catalog and its growing influx of live events and podcasts, the company built its own multimodal content embedding model, MediaFM.

MediaFM is the first tri-modal model pretrained using a portion of Netflix's catalog, jointly leveraging audio, video, and text to learn shot-level representations within movies or episodes. At its core is a Transformer-based encoder, with a focus on capturing the temporal relationships and narrative structure between shots.

The input is organized around shots segmented via shot boundary detection. For each shot, three embeddings are generated:

• Video: encoded using Netflix's internal model SeqCLIP
• Audio: encoded using Meta FAIR's wav2vec2
• Timed text: encoded using OpenAI's text-embedding-3-large, covering subtitles, audio descriptions, and closed captions

These three embeddings are concatenated to form a 2304-dimensional fused embedding, which is fed into the Transformer as a chronologically ordered sequence of up to 512 shots. Title-level metadata is injected via a [GLOBAL] token to provide overall title-level context as well.

The model architecture is a BERT-like encoder. First, the fused shot embeddings are projected to the hidden dimension, a learnable [CLS] token is prepended, and the title-level embedding is projected and inserted as a [GLOBAL] token following it. The sequence then passes through positional embeddings and the Transformer stack to produce contextualized representations, which are finally projected back to the 2304-dimensional space for prediction.

The training objective is Masked Shot Modeling (MSM). In each sequence, 20% of shots are randomly replaced with a [MASK] embedding, and the model is trained to reconstruct the original fused embedding. The loss minimizes the cosine distance between the predicted and ground-truth embeddings. The authors used Muon for hidden parameters and AdamW for the remaining parameters, noting that performance improved noticeably after switching to Muon.

Evaluation was conducted via linear probing, adding a task-specific linear layer on top of the frozen representation. Notably, for clip-level tasks, extracting embeddings within a larger sequence worked better than encoding the clip alone, indicating that context from surrounding shots substantially affects clip understanding.

Representative tasks include:

• Ad relevancy: multilabel classification to select clips suitable for ad placement, measured by Average Precision
• Clip popularity ranking: predicting relative popularity based on CTR, evaluated with Kendall's tau
• Clip tone: classification across 100 tone categories, using micro Average Precision
• Clip genre: classification across 11 genres, using macro Average Precision
• Clip retrieval: classifying whether a clip is clip-worthy, measured by Average Precision

As a result, MediaFM outperformed Netflix's internal SeqCLIP, Google's VertexAI multimodal embeddings, and TwelveLabs' Marengo 2.7 embeddings across all tasks. The improvement was particularly larger for problems like ad relevancy where surrounding context matters, and the authors concluded that contextualization, rather than multimodal fusion itself, was the key driver of the performance gains.