AI BriefingCookie Run: Kingdom Database Storage Layer Recovery Story
Key point
This covers the CockroachDB cluster outage caused by a script error right after Cookie Run: Kingdom's launch, and the recovery process that followed.
Details
Cookie Run: Kingdom experienced a service outage lasting about 36 hours, just 4 days after launch. The cause was a configuration error in a script intended to create Ballast files (dummy files) to prepare for potential disk space shortages.
The script pointed to a Block Device instead of a file path, resulting in an accident that overwrote a partition. This destabilized the file system of the AWS EC2 instances, causing 16 out of the total 24 nodes—a majority—to become isolated.
Because CockroachDB uses the Raft consensus algorithm, a majority of nodes must remain healthy for the cluster to stay operational. However, with more than half of the nodes failing, the issue escalated into a cluster-wide outage that paralyzed the entire system.
Upon recognizing the outage, the team immediately launched an emergency response and worked together with AWS and CockroachDB technical support teams to develop a strategy for restoring the cluster's state.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
GS Retail Builds Enterprise-Wide AI Gateway in Collaboration with AWS, Showcasing Automation of Authentication, Routing, and Account ManagementAWS Introduces Enhanced Custom Event Buses for Organization-Wide Event Sharing
KT Cloud Aims to Cut Deployment Failure Root Cause Analysis Time by 90% Using MCP