Kakao KHP Reveals Integrated Monitoring System for Metrics and Logs Across 100+ Clusters

Kakao operates the KHP Monitoring and Alerting System, which combines open-source tools with in-house developed features to manage over 100 clusters. To address the limitations of traditional metric-centric monitoring, this system adopts a dual-structure approach that continuously collects and analyzes both metrics and logs.

Metric Collection and Architecture

KHP collects various metrics in real time, ranging from host-level indicators to service-level indicators for services such as HDFS and HBase. Apache Druid was selected as the final storage solution due to its superior scalability and time-series processing performance, and the in-house developed KHP Agent was installed on each server for data collection and cleansing.

Existing open-source tools like Jolokia and Metricbeat failed to meet KHP's requirements, such as cumulative value correction and formatting for Druid, leading to the decision to implement a custom solution. The KHP Agent distinguishes between absolute values, such as CPU usage, and cumulative values, such as HDFS numOps, applies corrections accordingly, and transmits the data to Druid's real-time pipeline via Kafka.

Visualization and Log Processing

Grafana is used as the primary visualization tool, while Turnilo, based on Druid, is utilized for immediate data exploration when drill-down capabilities are required. For log processing, the Elasticsearch, Logstash, and Kibana (ELK stack) were adopted.

Specifically, the log processing architecture involves sending data collected by Filebeat to Elasticsearch via Kafka. This structure leverages Kafka's offset management features to ensure stable data ingestion without log loss and to minimize search latency in large-scale log environments. Additionally, dedicated dashboards for GC logs were built to analyze log patterns specific to HBase and HDFS, enabling precise tracking of the root causes of application performance degradation.