Netflix's Container Scaling Bottleneck: A Mount Lock Problem Exposed on Modern CPUs

After switching to a new container runtime, Netflix experienced container startups being delayed by tens of seconds or hanging entirely on some nodes like r5.metal. The cause wasn't simply image size, but rather a surge of mount/umount operations that occurred when launching containers with 50+ layer images all at once.

The core of the problem was that containerd, in a user namespace environment, performs open_tree(), mount_setattr(), and move_mount() for each layer to create idmap bind mounts, builds the overlayfs rootfs, and then cleans it all up again. Under the condition of 100 containers with 50 layers each, a total of 20,200 mount operations occurred just in the startup path, and all of these operations hit the kernel VFS's global mount lock, causing lock contention to explode.

This phenomenon was also explained by the difference from the older runtime. The previous approach had all containers share a single host user range and pre-translate UIDs at untar time, but the new runtime assigns each container its own unique host user range and efficiently maps ownership using the kernel's idmap feature. Security improved, but at the cost of higher concurrency overhead on the mount path.

In benchmarks, r5.metal was the first to fail under high concurrency, while m7i.metal-24xl and m7a.24xlarge held up much more stably. At low concurrency the results were similar, but as concurrency increased, the 7th-generation instances showed lower launch time and higher success rates, with m7a in particular showing the most consistent scaling.

The hot path identified through perf and custom microbenchmarks was Linux VFS path lookup — specifically a spin loop inside path_init() that repeatedly executes pause while waiting on a seqlock. Intel's Topdown Microarchitecture Analysis (TMA) showed that 95.5% of pipeline slots were tied up in contested accesses, and 57% was attributed to false sharing, revealing cache line bouncing and lock contention as the main culprits.

From a hardware perspective, two factors were especially significant. On the dual-socket r5.metal with NUMA, accessing the global lock added remote memory hops, increasing latency further; on m7i.metal-24xl with hyperthreading enabled, two threads competing for shared execution resources worsened lock contention. In fact, turning off hyperthreading improved container launch latency by 20-30%.

Ultimately, this wasn't simply a matter of containerd optimization — it was a bottleneck created by the combination of a global lock + modern CPU architecture + NUMA + hyperthreading. Netflix confirmed that container scaling bottlenecks need to be sought not in the software layer, but at the boundary between hardware and the kernel.