Evaluating Netflix Show Synopses with LLM-as-a-Judge

Netflix introduced an LLM-as-a-Judge approach to validate hundreds of thousands of show synopses at scale. The goal is to automatically judge the quality criteria defined by the creative team while also capturing signals that connect to actual member reactions.

Synopsis quality is split into two axes.

• Creative Quality: evaluates whether it meets internal writing guides and rubrics
• Member Implicit Feedback: measures member reactions via Take Fraction and Abandonment Rate

To build the creative quality criteria, three experts first repeatedly evaluated about 1,000 diverse synopses, and the criteria were refined through multiple rounds of calibration. In this process, using a binary score, referencing past examples, and maintaining a searchable taxonomy of common errors proved effective in raising agreement. After about 8 calibration rounds, writer agreement rose to about 80%, ultimately producing a golden set of about 600 examples.

The evaluation system then used separate judges per criterion. Handling all criteria with a single prompt degraded performance, whereas feeding in criterion-specific metadata and guides, generating explanations via zero-shot chain-of-thought, and then making a binary decision at the end worked better. APO (Automatic Prompt Optimization) was also used to refine prompts on a dev set of about 300 samples.

Two methods were also used to improve reasoning quality.

• Tiered rationales: reason at length, but summarize briefly before the final answer
• Consensus scoring: average results from multiple samplings to reach the final judgment

For example, on the tone criterion, tiered rationales boosted accuracy from 86.55% to 87.85%. 5x consensus also showed clear improvement on tone and clarity, but had little effect on criteria with short rationales like precision. Reasoning models also showed higher performance, but were excluded from the final system due to cost.

Agents-as-a-Judge was applied for factuality verification. Synopsis errors were narrowly evaluated by splitting them into the following four categories.

• Incorrect plot information
• Incorrect metadata
• Incorrect on- or off-screen talent
• Incorrect award information

The key was not to load too much context and criteria at once. Accuracy was more stable when each agent handled only one narrow factuality issue, and the scores obtained this way also showed correlation with core streaming metrics. As a result, Netflix is now able to use this system to identify and fix problematic synopses weeks or months before public release.