Building Kurly's Own MLOps - Kubeflow Adoption Story

Kurly's Data Platform team adopted Kubeflow as its MLOps platform to reduce the infrastructure burden on data scientists and create an environment where they can focus on modeling. Previously, they deployed and used JupyterHub, MLFlow, Airflow, and other tools on an EKS cluster, but the lack of connectivity between tools and growing complexity created a need for integrated management.

SageMaker vs Kubeflow Selection Criteria

Kurly conducted a comparative analysis of AWS SageMaker and Kubeflow. SageMaker offers natural integration with AWS resources and additional features like Ground Truth and Data Wrangler, but the possibility of vendor lock-in and high cost were pointed out as drawbacks. Kubeflow, on the other hand, offers high portability and scalability based on Kubernetes, and since Kurly was already actively using Kubernetes, being able to adopt it without separate infrastructure setup became the deciding factor.

Production Environment Optimization and Security Enhancement

Since Kubeflow's default installation method was insufficient for a production environment, Kurly performed custom configuration. To improve workflow convenience for data scientists, they adopted Elyra to convert code written in Jupyter Lab into KFP (Kubeflow Pipelines) DAGs. They also changed the artifact storage from MinIO to S3 and applied IRSA (IAM Roles for Service Accounts) to implement secure communication between pods and S3.

On the security side, they adopted SealedSecrets to prevent plaintext credentials from being exposed in manifests. This allows secrets to be managed in encrypted form in the GitHub repository, and a pre-commit git hook was used to prevent plaintext secrets from being accidentally pushed. Going forward, they plan to advance their MLOps environment through GitOps-based manifest management and hyperparameter tuning using Katib.