Kakao Adopts Orchestrator as Standard MySQL HA Solution to Replace MHA

Kakao has transitioned its existing MHA-based standard configuration to an Orchestrator-based one to ensure high availability (HA) in large-scale MySQL operating environments. MHA, developed in 2012, has had no patches or feature additions for over 10 years, making it impossible to support the latest MySQL and OS versions, and its Perl-based implementation made maintenance difficult. Additionally, it frequently malfunctioned during network failures and lacked the flexibility to configure HA based on Replica server status.

Selection and Configuration of Next-Generation HA Solution

Kakao conducted R&D starting in 2023 and reviewed 13 candidate solutions, including MySQL Cluster, ProxySQL, and Percona XtraDB Cluster, ultimately selecting Orchestrator. Orchestrator is an open-source project developed by Shlomi Noach, focusing on intelligently understanding and healing complex Replication structures. To ensure the high availability of Orchestrator itself, Kakao adopted a cluster configuration using the Raft algorithm. The Raft HA configuration consists of an odd number of nodes, with at least three, preventing Split-Brain phenomena through leader election and ensuring data consistency.

Custom Development and Operational Optimization for Kakao

Kakao customized Orchestrator to fit its internal DBaaS environment and built an integration system called Protego. Protego provides integration features between Orchestrator and the Kakao DBaaS environment and manages DB metadata. Additionally, Kakao independently developed a DNS Failover feature that monitors Replica server status and reroutes traffic via domain changes during failures. This feature changes the domain to another Replica within approximately 5 minutes of a Replica failure to ensure service continuity.

Changes in Operations and Expected Benefits

With the introduction of Orchestrator, the failure detection logic for MySQL has changed. Orchestrator determines server status based on the results of the Select 1 query and information from SHOW REPLICA/SOURCE STATUS. Accordingly, Kakao optimized the slave_net_timeout system variable value, which affects Failover time during network failures. Kakao is currently operating MHA and Orchestrator in parallel and plans to integrate all systems into Orchestrator within 3 years. Through this, Kakao expects to reduce Failover time caused by network failures and improve monitoring and management efficiency in large-scale MySQL environments.