MinIO Adoption Journey - Understanding HA and Establishing a DR Strategy

Critical data used from ML training to serving was separated from Ceph and moved to MinIO-based Object Storage. The targets were training datasets (DVC), model lineage (MLflow artifact storage), model weights for serving (MLflow model registry), and test data for CI/CD (S3), and distributed MinIO was chosen for its common access protocol and scalability.

To handle failures, drive failure, node failure, and IDC-level failure were defined as separate levels, and the DR strategy was divided according to the level of read/write disruption each service could tolerate. Reflecting the fact that service impact was medium or lower and the data didn't change frequently, Backup & Recovery was ultimately adopted as the default strategy, with RPO set at 7 days in the worst case.

For the backup method, original data backup was preferred over volume snapshots. It was judged that distributed storage requires precisely matching the state at the point of recovery (number of servers, capacity, etc.), and that behaviors such as write handling or locking during a snapshot can be framework-dependent, making recovery more complex.

During the verification stage, MinIO's Erasure Coding and Failover were directly checked. Testing was done by storing data across 4 servers with 2 data blocks + 2 parity blocks, and it was experimentally confirmed—consistent with MinIO's documentation—that when the total number of disks is N, read is possible if N/2 or more are alive, and writing a new object is possible only when N/2 + 1 are alive.

In the server failure test, data accessibility and consistency were checked under scenarios assuming 1 to 3 server failures. With 3 server failures, read operations were not possible with only 2 servers alive, and dashboard access—which also triggers writes—was likewise impossible with 2 or more failures. Conversely, after 2 server failures followed by a restart, access was only restored once 3 or more servers returned to normal.

In the drive failover test, Pods and PVCs were deleted and then redeployed to check the automatic recovery flow. mc admin heal ran in the background, recovering data while maintaining reads and writes, confirming that MinIO's recovery mechanism actually works. However, if N/2 or more fail or the damage is severe, recovery is not possible with MinIO alone, in which case all resources must be redeployed and then restored from backup data.

Scalability was also examined. For Scale Up, it was sufficient to remove the StatefulSet as orphan, increase the PVC capacity, and roll out—and scaling occurred without any read/write disruption even while a 30GB file was being copied. Scale Out, on the other hand, involves adding server pools, and read/write disruption occurred during testing; it was also confirmed that MinIO does not provide automatic rebalancing between pools.