Yogiyo Chaos Engineering (1): Starting Chaos Experiments

Chaos engineering was started on Yogiyo's membership-yo and stage-integration environments. The scope of experimentation was defined by first narrowing down realistically possible failures, targeting a service that was relatively low in operational impact and recently developed.

There were two scenarios.

• Gradually increasing pod network latency to check the service's response
• Blocking external API communication to verify the withdrawal flow and DB consistency

AWS FIS was chosen as the tool. This was because the target service was running on AWS EKS, and FIS clearly supported network latency injection. The monitoring metrics were set as latency, changes in pod count, and 5xx HTTP status codes.

During experiment preparation, two problems surfaced. First, the FIS pod with Istio applied was terminated right after startup because it couldn't communicate with the Kubernetes API server. The solution was to add a sidecar.istio.io/inject: false annotation to the FIS pod to block Istio sidecar injection.

Second, the temporary container injected into the pod needed root privileges for fault injection, but the service enforced non-root execution. Without touching the production environment, the issue was resolved by applying runAsNonRoot: false, runAsUser: 0, privileged: true, and allowPrivilegeEscalation: true only to the stage-only PodTemplate.

Part 1 focuses on how the chaos experiments were started and what trial-and-error issues were resolved. The actual observation results and insights will continue in Part 2.