Implementing 503-Error-Free Canary Deployment with Argo Rollouts in an EKS+ALB Environment

Deployment rolling updates make immediate rollback difficult, and the existing Blue/Green and default Canary approaches caused about 30 seconds of 503s during Promote in an EKS+ALB environment. This happens because at the moment of Promote, the Service selector changes, causing the EndpointSlice and ALB Target Group to be re-registered, and the new target remains in initial state before passing the health check.

The AWS Load Balancer Controller's readiness gate was also not a solution. Since this gate is only injected at Pod creation time, it doesn't apply to Blue/Green and default Canary, which change the selector of an already-running Pod later.

As a solution, we applied Canary PingPong from Argo Rollouts v1.2. By fixing two Services, pingService/pongService, only the stable/canary roles alternate with each deployment, and at Promote time, instead of changing the selector, only the ALB ForwardConfig weight is swapped. As a result, target re-registration doesn't occur, and the 503 window disappears.

The application points are as follows.

• 3 Services: rootService, pingService, pongService
• Ingress backend: name: use-annotation
• Rollout: pingPong, trafficRouting.alb, maxUnavailable: 0, maxSurge: 1
• Deployment stages: setWeight, pause
• Optional features: setCanaryScale, analysis

In operation, it's important to allow sufficient observation time at each stage, and the approach of manual Promote using pause: {} can also be utilized.