Why QA Killed the Servers - Host Level Chaos Engineering Tests

Host Level chaos engineering is an experiment that injects failures not into application code, but into the infrastructure itself, such as DB, message queue, cache, and search engine, to verify resilience.

OliveYoung QA performed both full-block tests and failover tests together in a QA environment identical to production, and directly verified actual screens, order history, and data consistency based on customer scenarios and admin scenarios.

From the customer's perspective, they followed the cart, coupon, order, and payment flow to check which screens became slow or unresponsive during a failure. From the admin's perspective, they checked whether data was properly reflected after recovery following product edits, gift promotion registration, coupon creation, and banner changes.

During testing, they found 9 major bugs, of which 4 were fixed immediately, and the remaining 5 were incorporated into the 2026 roadmap. At the same time, they built a failure detection alert system so that the responsible team can recognize and respond as soon as a failure occurs.

There were three key patterns.

• The system may be alive, but the customer experience can still break down: cases occurred where payment completed without the gift item during a DB failure, or where payment finished but the customer saw a 504 error during a message queue failure.
• Cache is a 5-minute shield: thanks to the TTL cache, the system held up briefly, but after 5 minutes, search became unavailable, category/brand pages showed errors, and the main screen slowed down.
• The real test is after recovery: coupons recovered automatically, but gift item stock increased abnormally, and product information required manual work, leading them to even build a dedicated API for post-recovery consistency work.

Based on these results, they newly established criteria for immediate action, recovery criteria, and priority criteria. In particular, they established the standard that everything from failure detection to recovery or failover switchover must be completed within 5 minutes for customers not to notice.

The first application case was the product detail system MSA migration project. When the DB went down, the circuit breaker appeared to work normally, but the QA screen revealed a problem where the circuit breaker was missing for the maximum benefit price section, causing the entire product detail page to fail to render.

Ultimately, the core of Host Level testing lies not in whether the server holds up, but in discovering how it breaks down and what breaks in unexpected ways first. QA takes on the role of being the first to confirm that even when system logs look normal, the customer-facing screen may already be experiencing a failure.