Kakao Unveils 'KHP', Its Own Hadoop Platform to Break Free from External Vendor Dependence

Kakao has unveiled its Kakao Hadoop Platform (KHP), a Hadoop platform developed in-house to address the high operational costs and difficulties in modifying features associated with external vendor solutions. Since the emergence of cost risks for Hadoop solutions in 2019, Kakao has been directly building and operating the platform to ensure flexibility and cost efficiency in managing large-scale clusters.

Five Core Components of KHP

KHP consists of five components that cover the entire process from compatibility verification of the Hadoop ecosystem to deployment and operational management.

• khp-package: Management of distribution versions that have verified the compatibility of Hadoop components
• khp-ansible: A deployment tool responsible for cluster installation, configuration management, control (start/stop), upgrades, and migrations
• khp-agent: An agent that collects metrics from each host and supports Grafana dashboard configuration
• khp-server: Visualizes cluster status, controls processes, generates alarms, and manages Audit logs
• khp-log: A service supporting log ingestion and search

Operational Efficiency and Migration Strategy

khp-ansible simplifies installation and configuration deployment through a design that guarantees Idempotency, and enables rapid rollback during patch upgrades via a binary replacement method using symbolic links. Additionally, it supports In-place migration, which replaces only binaries without physical data copying to facilitate the transition from existing solutions to KHP, significantly reducing time and costs.

Regarding monitoring, khp-agent transmits collected data to Druid, and an alarm system sends notifications via KakaoTalk or KakaoWork when thresholds are exceeded, thereby enhancing operational stability.