Kakao Reveals Part 2 of KHP Monitoring System: Rule-Based Alerts and Flexible API Implementation

Kakao's Hadoop operations team unveiled the monitoring and alerting architecture of the KHP system, explaining how they resolved flexibility and operational efficiency issues that existing commercial solutions failed to address. To mitigate the 'boy who cried wolf' effect and alert fatigue in an environment where a small team manages over 100 clusters, they built a central management server, KHP Server, which supports threshold adjustments, alert delays, batch sending, and channel diversification.

Flexible Monitoring Implemented in Code

Instead of simple table-based rule definitions, KHP adopts an approach where monitoring logic is written in Ruby-based code. This is to meet complex operational requirements such as arithmetic operations across multiple metrics, applying different thresholds per cluster, and branching on non-metric conditions. Developers can query Druid metrics via high-level APIs like all, max, and sum, perform operations on them as scalar values, and compare them against thresholds. Additionally, the w (warning) and c (critical) methods reference thresholds configured per cluster, enhancing management convenience.

Log-Based Monitoring and Alert Management

Log monitoring is implemented by abstracting Elasticsearch query processes into high-level APIs, treating the frequency of specific log patterns as metrics. Using APIs like log_count, issues can be detected with conditions like 'if present' or 'if frequent' without complex queries. Collected issues are sent through various channels such as KakaoTalk and KakaoWork, with the ability to expand channels via plugins. All dispatch histories are stored in Elasticsearch for statistical analysis and checking of inappropriate rules.