Curry Builds Daily Automated Deployment Pipeline for Recommendation Models Using Argo Workflows and SageMaker

• Curry's AI Search & Recommendation Development Team built a pipeline that automates the process from training recommendation models to switching them into service by combining Argo Workflows and AWS SageMaker
• Applied a generation-based switching structure to bundle inference endpoints, user embeddings, and search indexes into a single set for simultaneous replacement
• In A/B tests conducted across four surfaces from mid-July to late August, the add-to-cart click rate increased by 3–13% depending on the surface
• Improved product discoverability led to an average increase of approximately 8% in the number of clicked products per user and an average increase of approximately 9% in total clicks

Architecture and Tool Selection

• Adopted a hybrid structure where Argo Workflows, a general-purpose workflow engine, handles orchestration, and SageMaker, a managed service, handles the GPU training and inference runtimes
• Used SageMaker's Model Registry to manage model approval status and leveraged the Weight of ProductionVariant to pre-deploy new models with 0 traffic
• Used the Inference Component to compare and validate the champion (current service) and challenger (new candidate) models on the same endpoint

Generation Switching and Rollback Strategy

• Switched generations by changing only the sdm:active key in ElastiCache, while retaining the previous generation rather than deleting it immediately to allow for rollback
• Discovered that calling UpdateEndpoint replaced the entire endpoint Fleet, and added warm-up logic to execute a Forward pass immediately after model loading to prevent latency
• Designed the pipeline to ensure idempotency even upon failure during execution, so that re-runs skip already completed steps and resume from the interrupted point