Data Canary: How Netflix Validates Catalog Metadata

Netflix's catalog metadata is core data that determines whether content exists, whether it's playable, and more. Recently, an incident occurred where a data feed error—without any code changes—caused metadata for some titles to go missing, resulting in playback failures. This exposed the limitation that the existing Canary approach for code deployments could not detect corruption in the data itself.

To address this, Netflix introduced the Data Canary Orchestrator Pattern. This system was designed to overcome the temporal constraints of existing code canary tools and the limitations of Shadow Traffic.

Key features include:

• Dedicated Orchestrator Pattern: A separate orchestrator instance synchronizes the state of the Baseline cluster and the Canary cluster, coordinating the canary flow.
• Real Production Traffic: Instead of shadow traffic, real production traffic is used to validate actual customer impact across the entire playback lifecycle.
• Rapid Detection: Issues are detected within the data cycle, and if a problem occurs, deployment is immediately blocked to minimize the Blast Radius.

Through this system, Netflix detects errors occurring during data transformation within 10 minutes, preventing faulty data from being delivered to users worldwide.