Postmortem: January 2022 100% Discount Event Outage

The 100% discount event held in early January 2022 caused 5 to 20 times the usual traffic, resulting in a severe outage of the Inflearn service. This outage was not simply due to a traffic surge, but was compounded by technical debt and design shortcomings.

The main causes of the outage are as follows.

• Connection exhaustion due to slow queries: In the interested companies lookup feature on the course page, a query without an applied index was executed, occupying all connections of the Node.js process. This caused the process to be forcibly terminated, paralyzing the entire service.
• Insufficient DB load balancing design: To reduce the load on Redis, some cache data was moved to AWS Aurora PostgreSQL, but there was a structural flaw where all cache data was stored in a single JSON Row. In the process of reading and writing this massive Row, the CPU usage of the write DB spiked to over 90%, triggering the outage.

Through this outage, Inflearn identified its technical debt, and intends to transparently share its process of resolving the issue through measures such as adding indexes and improving the cache structure.