Service Discovery DR Configuration Part 3 - Exploring Spring Cloud LoadBalancer for Zone Failover in eurekube-operator

11st's Vine Platform is a Spring Cloud-based MSA platform running about 720 instances and 70+ applications, and it has expanded into a Hybrid Cloud structure that uses IDC and AWS together.

eurekube-operator is a Kubernetes Operator that ties together Client-side Service Discovery in IDC and Server-side Service Discovery in EKS. Previously it operated centered on the IDC Eureka Server, but after configuring the Eureka server as Multi-Zone, it became necessary to have it prioritize the EKS-side Eureka Server and support Zone Failover that naturally switches to IDC in case of failure.

The core idea is to layer failover logic on top of Spring Cloud LoadBalancer. The default-provided ReactorLoadBalancerExchangeFilterFunction only selects an instance and sends the request, and you need to enable spring.cloud.loadbalancer.retry.enabled = true to make use of RetryableLoadBalancerExchangeFilterFunction. Load balancing itself is handled by ReactiveLoadBalancer and ServiceInstanceListSupplier, while pre/post request hooks are handled via LoadBalancerLifecycle.

The problem is that eurekube-operator sends requests directly via WebClient rather than through a Eureka Client, so it cannot use the zone switching and recovery mechanisms that the Eureka Client provided, such as RetryableEurekaHttpClient and SessionedEurekaHttpClient. So a separate implementation is designed to satisfy the following two requirements.

• If all instance requests in a specific Zone fail, switch to a different Zone.
• After a certain amount of time passes, clear the failure history and return to the original Preferred Zone.

For this, a Caffeine cache is set up to store failed ServiceInstances for a TTL. The example configuration uses a 20-minute TTL, and it's configured to return an empty set on a cache miss. After that, ZoneFailoverAwareLoadBalancerLifecycle records failed instances into the cache, and ZoneFailoverAwareCachedInstanceSelector recalculates candidate instances by considering both the cache and the Preferred Zone together. Finally, ZoneFailoverAwareLoadBalancer selects one of those candidates using a RoundRobin approach.

In summary, when the EKS-side Eureka Server is healthy, that Zone is used preferentially; when a failure or high latency occurs, it switches over to IDC; and after time passes, it returns to the Preferred Zone again. In Chaos Tests, it was confirmed that the intended Zone Failover works correctly in both server abnormal termination and network delay scenarios.