Cookie Run: Kingdom Guild Update Service Issue Retrospective

CockroachDB splits and stores data in units called Range, and a phenomenon called Hot Range (Hot Spot) can occur when load concentrates on a specific Range.

Right after the guild update in March 2021, CPU usage on a specific node spiked, causing service errors. This happened because the load was not evenly distributed and excessive load concentrated on a specific node.

The root cause was the Prefix structure of the index combined with a change in the user inflow pattern. Previously, user inflow was distributed across tutorial stages, allowing natural data splitting (Split) to occur, but the guild update had almost no inflow funnel, causing write operations to surge on a specific Primary Key range.

Additionally, during code refactoring, the Primary Key format was changed, resulting in only a very small portion of the previously split Ranges being used, which worsened the Hot Range phenomenon.