Kakao Releases Open-Source HBase Network Packet Analysis Tool 'hbase-packet-inspector'

Kakao has released hbase-packet-inspector (HPI), an open-source tool that analyzes network packets from HBase region servers to extract request and response information. It was developed to identify detailed access patterns and performance bottlenecks that were difficult to detect with existing monitoring tools.

Background and Necessity

Kakao operates dozens of HBase clusters, managed by more than five personnel. Detailed workload issues, such as excessive access to specific row keys or the concentration of increment operations in specific regions, were difficult to identify with existing metrics, necessitating a precise analysis tool at the network level.

SQL-Based Analysis and Architecture

HPI loads captured packet data into a built-in H2 database in memory, allowing users to flexibly query the data using SQL queries. This enables analysis based on various conditions, such as 'list of most accessed row keys' or 'average response time for a specific region.' It includes a built-in web-based SQL interface, making it immediately usable without separate UI development.

Kafka Integration and Scalability

To overcome memory-based limitations and enable real-time analysis across the entire cluster, Apache Kafka transmission functionality was added. HPI running on region servers or application servers sends data to Kafka, which Kakao processes using Spark streaming, loads into the Druid OLAP engine, and visualizes with Pivot to enhance hotspot analysis and automated response processes.