Yogiyo Chaos Engineering (2) Summary of Chaos Experiment Results

In a Stage environment, scenario traffic of about 350 RPS was continuously injected using Locust, and Pod network latency and external API communication failures were each injected to check service resilience.

In the Pod Network Latency experiment, 500ms latency created the biggest impact. Pod CPU dropped from 0.1~0.2 core to 0.01~0.03 core, RPS nearly converged to 0, and P99 request latency spiked up to 30 seconds max. App logs showed HTTP 500 along with errors that appeared to be Redisson / Netty thread resource exhaustion.

Traffic flowed through the path Locust → ALB → Istio IngressGateway → service Pod, and Istio generated a large number of UH (Unhealthy Upstream Host) responses. In other words, since there was no healthy upstream, the gateway immediately returned 503, and as a result, Locust's RPS fluctuated significantly, oscillating between fast failure responses and very slow success responses.

In contrast, 250ms latency showed a different pattern. RPS remained relatively stable at around 17, and while P99 increased to about 5 seconds, Pod status was maintained. There were no App errors either, and the Happy Path Test passed, confirming the range where "latency is large but the service holds up."

The key insights gained from this experiment are as follows.

• More intuitive exception handling is needed to surface Redisson Error.
• There is room to improve asynchronous processing to prevent subscription member order failures during membership service outages.
• The bottleneck was Queue occupancy at the Redis connection stage, requiring reinforcement of Queue management logic.
• Rather than simple Health Checks, the impact of Pod Latency is revealed much more significantly under scenario-based traffic.
• AWS FIS is useful, but clear constraints emerge when combined with organizational permissions and governance.

The second experiment verified external API communication failure. When the path calling the external partner's wiithdraw-subscription API was blocked via AWS NACL, the error rate of the withdrawal API surged, and while the App returned 500, consistency of DB fields such as Started_at / Finished_at was maintained. In other words, while the member withdrawal feature broke, other membership features operated normally, confirming a certain level of resilience against external vendor failures.