Troubleshooting GPU nodes repeatedly replaced due to DCGM errors on EKS Bottlerocket AMI

Inflearn operated a GPU cluster for lecture video upscaling using a combination of Amazon EKS 1.32, Bottlerocket AMI (amazon/bottlerocket-aws-k8s-1.32-nvidia-aarch64-v1.39.1-3a880b44), g5g.xlarge GPU instances, and Karpenter v1.4.0. Even with Node Auto-Repair (nodeRepair: true) enabled to prevent zombie node states, they ran into a problem where nodes were judged abnormal and continuously replaced about every 10-11 minutes after a GPU Pod was launched.

The key clue was the AcceleratedHardwareReady status in the node events. Right after a node became Ready, the log showed Status: True → False, Reason: DCGMError, Message: failed to initialize DCGM: libdcgm.so not Found, meaning that the missing NVIDIA DCGM library made it impossible to properly check the GPU hardware status. GPU computation itself wasn't blocked, but because the DCGM-based health check failed, Karpenter treated the node as unhealthy.

Karpenter then checked the node pool's health status and proceeded with replacement based on the unhealthy condition and the policy's termination time. The post summarizes Karpenter's logic as follows:

• isNodePoolHealthy() checks the ratio of unhealthy nodes within the node pool
• findUnhealthyConditions() detects conditions such as AcceleratedHardwareReady=False
• After a grace period based on policy, it rechecks via RequeueAfter
• Once the grace period elapses, the NodeClaim is deleted and the node is terminated

Ultimately, the root cause of the issue was that it's difficult to directly patch in libdcgm.so in the Bottlerocket environment. Switching to a different AMI was also an option, but the infrastructure migration cost was high, so disabling Node Auto-Repair (nodeRepair: false) was chosen as the most practical solution for the current situation. The post concludes that if GPU nodes on Karpenter experience frequent replacements, one should first suspect AcceleratedHardwareReady and DCGM errors, and reconsider the Auto-Repair configuration if necessary.