Finding a Zombie in the System: A Real-World Case of a CPU Bottleneck

Pinterest's Kubernetes platform team tracked down an issue where Ray-based ML training jobs would intermittently lose network connectivity and crash. At first it looked like instability in distributed training, but the logs repeatedly showed ENA Network driver reset, pointing the team toward CPU starvation, a cause commonly cited in AWS documentation.

On the machines where network resets occurred, some showed elevated system CPU usage and page fault counts. The team tried mitigations such as TransparentHugePages, jemalloc, CPU affinity via taskset, and ENA interrupt pinning, but none reliably eliminated the problem. Rebooting only helped for about a week before resets started appearing again.

The problem was further complicated by the fact that it didn't appear uniformly across all zones. Even with identical Kubernetes versions and system images, resets were concentrated in us-east-1a, and a review with AWS support only yielded the conclusion that "something somewhere is hogging the CPU, starving the network thread."

At this point, instead of relying on high-level metrics, the team dug into the actual CPU-consuming code using profilers like perf and mpstat. This was an issue easy to miss when looking only at whole-machine average utilization — the key insight was that on a GPU node with 96 vCPUs, saturation of just a handful of cores could be enough to starve the network thread and trigger a driver reset.