Building a robust, high-performance ScyllaDB cluster with Super Disk

Hyperconnect has run ScyllaDB as its company-wide NoSQL database for about 4 years, meeting the high-performance demands of various services including an ML feature store. But a recent failure drill revealed that node recovery time during cluster rolling updates was too long.

The core of the problem was the nature of i4i instances using Local NVMe SSD. Unlike Cassandra, ScyllaDB doesn't rely heavily on the Linux page cache and instead depends more directly on its embedded cache and disk I/O performance, so when an instance reboots and the local disk disappears, the recovery process of re-fetching data from other nodes inevitably takes a long time. In practice, replacing a single node took 18-24 hours, and node recovery took 3-4 hours for cases with a lot of data, up to as much as 12 hours.

The solution introduced to address this was Super Disk. Referencing Discord's case, the setup binds Local SSD + EBS into a RAID1, with write-mostly configuration so that reads are performed only from the Local SSD. This way, EBS handles persistence while the Local SSD absorbs read latency.

The PoC was conducted by creating two clusters of the same spec, i4i.4xlarge, and comparing them.

• ScyllaDB version: 5.4.9
• pod CPU: 14
• pod memory: 110Gi
• scylla embedded memory size: 100G
• EBS capacity: 3492GB
• Initial loaded data: 700 million rows, about 1.5TB per node

Failure scenarios were also verified. When EBS was force detached, the RAID went into a degraded state, but read/write continued to work normally in [_U] status. Conversely, when the i4i instance itself was force-terminated and the EBS was attached to a new instance to rebuild the RAID, recovery took about 1 hour, and afterward restarting ScyllaDB let it rejoin the cluster normally. Since RAID recovery uses both EBS and Local SSD resources, it was more efficient to bring ScyllaDB up only after recovery finished.

Performance was maintained beyond expectations as well. Measured using YCSB and Latte with read/write ratios close to actual production, there was almost no difference in throughput, write latency, or overall latency between the cluster with Super Disk applied and the one without. There were brief spikes in read latency depending on compaction timing, but overall performance degradation was minimal.

In conclusion, Super Disk was a practical alternative that secured persistence, fast recovery, and performance similar to before all at once. Based on this assessment, we'll continue with a follow-up covering Windmill-based automation for managing Super Disk more efficiently in a Kubernetes environment.