Building an AI Platform: From GPU Adoption to Kubeflow

In the existing model serving structure, even after registration in MLflow, ML engineers had to manually create Container images and deploy them to production GPU servers. Repetitive manual work and environment differences placed a heavy burden on both speed and stability.

To solve this, Kakaopay chose Kubeflow based on standardization, scalability, and integration. Judging that Airflow or MLflow alone would have limits in long-term operation and scalability, they designed an on-premises AI platform suited to an environment already centered on Kubernetes-based operations.

The new platform separately supports two user groups.

• AI experts automate everything from training to model registration using Kubeflow Pipeline.
• Service developers simply call and deploy models through their own web console.

Afterward, to satisfy both performance and cost simultaneously, they introduced H200 GPUs, RDMA, and Infiniband, and built a hybrid cluster separating regular workers from GPU workers. On GPU nodes, they utilized GPUDirect to reduce CPU bottlenecks and improve the efficiency of large-scale distributed training.

To increase GPU utilization, they also applied MIG (Multi-Instance GPU). This splits a single physical GPU into multiple isolated instances, allowing multiple workloads to use it stably at the same time.

At the serving layer, they reviewed KServe and Knative, but Scale to Zero had a Cold Start problem that could take up to 6 minutes or more on the first request. Prioritizing service responsiveness, they decided to keep minReplicas at 1, choosing stability over cost efficiency.

Afterward, to use GPUs more economically, they established a partitioning strategy based on usage (training/inference), model used, and model size. Finally, they monitor total GPU resources and quotas, idle resources, and GPU usage to reduce waste and continuously improve the partitioning policy.