Kurly Builds Real-Time Serving Architecture for Cart Recommendation Service Based on TorchServe

Kurly's Data Service Development Team shared their experience building and operating a real-time serving architecture for a cart recommendation model. The overall system uses a mix of AWS and GCP, with main recommendation model tasks performed on an AWS EKS cluster. Training data is loaded into BigQuery, and monitoring is done via Datadog and Slack.

Serving Framework Selection and Performance Optimization

After comparing BentoML and TorchServe as serving frameworks, TorchServe was ultimately selected for showing better metrics in terms of response time. During the performance tuning process, it was confirmed that CPU Clock Speed is a key factor in achieving low latency. This is because TorchServe processes requests by placing them in a Queue and handling them in FIFO order, meaning fast processing of preceding requests has a significant impact on overall performance. Other options such as async_logging and netty_client_threads settings did not have a significant impact on latency improvement.

Applying GracefulShutdown for Zero-Downtime Deployment

To prevent downtime during deployment caused by frequent ML model updates, a GracefulShutdown strategy was introduced. In the AWS ALB Ingress environment, downtime can inevitably occur during TargetGroup disconnection and new connection processes. To solve this, terminationGracePeriodSeconds was set longer than the ALB idle timeout (60 seconds), and a PreStop Hook was used to secure wait time before pod termination. This setting was confirmed to prevent 502 errors and enable stable termination.

Warm up and MLOps Environment Setup

To solve the problem of over 3 seconds of latency occurring on the first request after deployment completion, a Warm up process using Kubernetes StartupProbe was added. This was to prevent class loading and model initialization from being performed at the time of the first request in TorchServe's Frontend (Java) and Backend (Python) structure. The MLOps environment was built based on Kubeflow Pipeline and MLFlow, and a GitOps-based deployment pipeline was automated using ArgoCD and ArgoCD Image Updater, separating the model repo and serving repo.