Kakao Unveils 'KEMI', a Company-Wide Resource Monitoring System

Kakao has unveiled KEMI (Kakao Event Metering & monItoring), a monitoring system for integrated management of company-wide resources. KEMI consists of KEMI-STATS, which collects metric data from servers and containers, and KEMI-LOG, which analyzes logs.

KEMI-STATS: Hybrid Collection and Resource Optimization

KEMI-STATS monitors tens of thousands of servers and containers, employing both polling and push methods in parallel. Physical servers and virtual machines collect data every minute via SNMP regardless of OS type, while containers are handled via push. Collected data is stored in OpenTSDB via Kafka and Samza and visualized with Grafana.

Notably, for resource efficiency, it offers the Cuota service, which automatically detects low-usage VMs and notifies responsible parties. This allows for the deletion or consolidation of unnecessary VMs to reduce infrastructure costs. It also features scalability that allows dynamic changes to collection items without restarting the Poller by leveraging etcd.

KEMI-LOG: Large-Scale Log Processing

KEMI-LOG processes logs on the scale of hundreds of gigabytes per day. Logs collected via Fluentd are distributed and stored in Hadoop and Kafka. Data stored in Hadoop is indexed into Elasticsearch via Hive batch jobs and can be queried in Kibana, while data stored in Kafka generates real-time alerts through KEMI Dike, which utilizes Storm and Redis.

Log alert rules are stored in etcd in standard SQL format and applied dynamically without restarting the Storm topology. Generated alerts are integrated with custom metrics from KEMI-STATS and can be monitored on an integrated dashboard.