Netflix Achieves >99.95% GPU Availability with Automated Reliability System

Netflix has implemented a four-layer reliability system for its GPU infrastructure to support large-scale training and inference jobs, achieving >99.95% availability of GPU nodes. The system addresses the fragility of gang-scheduled training, where a single hardware failure can halt an entire multi-week job, by detecting faults before and during execution and automatically remediating affected machines.

Why Basic Checks Fail

Standard cloud provider status checks and basic GPU driver verifications are insufficient for distributed training. A GPU can appear healthy while suffering from on-chip memory faults or degraded network bandwidth that only manifest under load. Furthermore, diagnostic tools themselves can hang or fail silently, creating false confidence. Netflix’s approach assumes hardware and inspection tools can lie, requiring checks that fail closed if dependencies are missing or unresponsive.

The Four-Layer Reliability Model

The platform treats reliability as four distinct layers, each catching what the previous layer misses:

• Node and Hardware Health: Device-level checks using NVIDIA DCGM and a node-level agent based on Kubernetes node-problem-detector run every 30 seconds. They inspect GPU error codes, interconnect integrity, thermal states, and network adapter link counts. A custom fatal-error map distinguishes between benign and critical codes across different GPU generations.
• Scheduler and Platform: Controllers watch Kubernetes node conditions to taint or drain machines. A graded remediation policy distinguishes between transient "problem" conditions (which block new work) and persistent "failure" conditions (which trigger evacuation and replacement). Karpenter node auto-repair is customized to tolerate flaky errors briefly while acting quickly on fatal ones.
• Training Loop: Jobs are designed to survive machine loss through checkpointing and elastic workers. The scheduler automatically retries platform-caused failures, but only after attributing the error to hardware rather than user code to avoid masking bugs.
• Observability: Metrics track availability, job start success, and job completion success, broken down by cause (hardware, driver, scheduler, user) to pinpoint system weaknesses.

Results and Future Improvements

The shift from manual intervention to automated, layered checks significantly improved fleet utilization and reliability. Key metrics include:

• >99.95% availability of GPU nodes.
• Job success rates increased from ~98% to above 99.5%.

Netflix is now extending active checks to cover more failure modes under load and implementing Kubernetes Dynamic Resource Allocation (DRA) to isolate faulty GPUs on multi-GPU machines rather than draining entire nodes. Future work includes using sched_ext for custom Linux scheduling to prioritize collective operations and data loading, reducing GPU idle time across large clusters.