Kakao Reveals Real-Time Messaging Performance Testing and Go pprof Bottleneck Analysis Techniques

Kakao's Interaction Platform team revealed techniques for performance test design of real-time messaging systems and bottleneck analysis using Go pprof. The core purpose of performance measurement is comparing resource efficiency, with subscriber count (N) and message publication volume (M) set as key variables to standardize test criteria.

Performance Test Environment and Scenarios

Real-time messaging involves tightly coupled Subscribe and Publish operations, making it difficult to define as simple request/response transactions. Tests were conducted in Kubernetes cluster and VM instance environments, with publishing clients accessing Redis directly to measure Fan-out performance. Subscribing clients reproduced actual request patterns in a container-based multi-process/thread environment.

Initial test results confirmed that subscriber count and message publication volume had the greatest impact on performance, surpassing other factors such as hardware resources or message size. Increasing subscriber count raises memory allocation and CPU usage, while increasing publication volume raises I/O operations and CPU usage, respectively.

In-Depth Analysis with Go pprof

Since simple metrics (such as reaching 100% CPU usage) were insufficient to identify specific bottleneck points, the pprof tool was introduced. CPU profiling is conducted via sampling, visualizing per-function CPU occupancy and call graphs through the top, list, and web commands.

Analysis cases confirmed characteristics such as sub-function time being included in the cumulative time (cum) of parent functions in nested call structures, and Total samples potentially exceeding Duration in multi-processor environments. Notably, analysis of the Caribe system identified that I/O operations during the Fan-out process from Publish to multiple Subscribes accounted for the majority of CPU usage, pinpointing this as the core bottleneck.