Multimodal Intelligence Behind Video Search

Video search can't be solved with simple keyword matching. Signals produced by different models—scenes, people, objects, dialogue—must be aligned onto a single timeline and turned into a structure that can be queried in real time.

To do this, Netflix separated the process into a 3-stage pipeline instead of searching raw model output directly. First, the annotation service persists original annotations at high speed to Cassandra, and then Kafka events trigger offline processing jobs.

Offline fusion splits continuous detections into 1-second temporal buckets, combining overlapping segments. For example, if the character "Joey" is detected from 2-8 seconds and the scene "kitchen" from 4-9 seconds, the 4-5 second segment becomes a single record combining both annotations, and this enriched record is stored back into Cassandra.

This structure maintains temporal consistency and write performance even as a 2,000-hour production archive grows to over 216 million frames. It then upserts using the same asset ID + time bucket as a composite key to create a single, deduplicated source of truth, and finally loads it into Elasticsearch to build a sub-second search index.

At the search stage, the query is first interpreted to perform query type detection, filter extraction, and vector transformation. Then, exact k-NN and ANN (HNSW), cosine similarity, and Euclidean distance are combined as appropriate to quickly return frame-level results that reflect both text conditions and vector embeddings together.

The key lies not in a single model but in unifying the outputs of multiple specialized models under a common representation of timeline, text, and vectors. This frees exploration from slow manual work, letting creators find the exact moment they want almost instantly, even within complex video archives.