The Journey to Auto-Scaling EKS Node Groups for Job Workloads

Karrot's Cluster team has operated a dedicated EKS Node Group, separate from server workloads, because Job workloads are hard to interrupt once they've started. However, relying solely on a fixed number of nodes caused delays when jobs piled up, and wasted cost during idle periods, creating significant operational burden.

Applying auto-scaling required two things. The first was bin-packing, packing Pods onto as few nodes as possible, and the second was a mechanism to prevent running Jobs from being forcibly terminated during scale-in.

Bin-packing was implemented using PodAffinity instead of a separate scheduler. Pods with the group: job label were guided to be placed on the same node, and this was applied in bulk to Argo Workflow Pods using Kyverno. In testing, Pods that had previously been evenly spread across multiple nodes were confirmed to be concentrated on a subset of nodes after applying PodAffinity.

To prevent forced termination, cluster-autoscaler.kubernetes.io/safe-to-evict: "false" from Cluster Autoscaler and karpenter.sh/do-not-disrupt: "true" from Karpenter were used. Additionally, it was noted that when using Cluster Autoscaler with a Multi-AZ ASG, AZ rebalancing could disrupt Job Pods, so it needed to be turned off.

It worked normally in the Alpha environment, but in Production, new problems emerged as large volumes of Job Pods piled up every hour on the hour. As Pods concentrated on fewer nodes, the load per node spiked sharply, and the following issues occurred in particular.

• kubelet overload: The kubelet_pleg_relist_duration_seconds_bucket metric spiked, and because the Pods' CPU requests were lower than actual usage, even more Pods were packed onto a single node.
• Image Pull failures
• EBS Volume throttling
• CNI Plugin IP assignment delays

The author explains that increasing kube-reserved was also tried, but it wasn't sufficient without Pod right-sizing. The key conclusion is that bin-packing alone is not enough, and the operational load caused by node concentration must also be controlled together.