Building Kurly's Own MLOps - Laying the Foundation

Kurly developed its own MLOps platform using its existing Kubernetes-based infrastructure, setting automatic Provisioning and Deprovisioning of GPU resources as a core task. In the existing environment, tools operated independently and there was no integrated work environment, and since GPU costs were expensive in the cloud environment, manual monitoring and permission requests by engineers were essential.

Background and Structure of Karpenter Adoption

Kurly reviewed integrated environments such as AWS SageMaker or GCP VertexAI but excluded them due to cost issues, and instead chose Karpenter, which utilizes the existing Kubernetes infrastructure. Karpenter is an open-source project created by AWS that handles automatic scaling of Kubernetes Worker Nodes.

The existing Cluster Auto Scaler(CA) manages node groups using EC2 Auto Scaling Group(ASG) and Launch Template, so it has many steps and slow scaling speed. In contrast, Karpenter operates independently of the Cloud Provider, continuously observes Unschedulable Pods, and directly performs Worker Node deployment and deletion when needed. It also handles requests to bind Pods to specific Worker Nodes in place of kube-scheduler, enabling much faster automation compared to existing ASG.

GPU Resource Management and Deprovisioning Optimization

GPU server provisioning is performed by including GPU requirements in Karpenter's Custom Resource, Provisioner. However, immediately after a GPU node is created, there are no NVIDIA-related settings so it cannot be used right away, so the NVIDIA device plugin for Kubernetes(NVDP) must be installed as a daemon set to detect GPU-equipped nodes and quickly install the plugin.

To solve the Deprovisioning problem where nodes are not deleted when GPU usage stops, Node Tags and Labels were utilized. Through Anti Node Affinity or Node Selector, pods unrelated to GPU were blocked from being allocated to GPU nodes. Also, starting from Karpenter v.0.15.0, the Consolidation option, which replaces workloads with cheaper nodes or removes unnecessary nodes, can be enabled to reduce cluster costs.