Kakao Develops Proprietary Log Collector 'Logriver' to Replace Fluentd

Kakao has introduced Logriver, a proprietary log collector developed to replace Fluentd, the main component of its internal log collection system kemi-log. The existing Fluentd had structural limitations, including the lack of large-scale cluster management features, the need to restart processes when changing configurations, and high memory usage.

Logriver was developed in the Go language and supports zero-downtime configuration changes through a central controller and agent architecture. By leveraging Go's goroutines and SO_REUSEPORT, it allows dynamic configuration updates without process restarts. Additionally, the go-json library was applied to optimize data encoding/decoding performance.

The key achievements after adoption are as follows:

• Memory Usage: Significantly reduced from 90% to 15%
• Network Connections: Reduced the number of Kafka connections from 4,000 to 200
• Server Resources: Cut total server usage by 50%
• Operational Stability: Mitigated DDoS attacks and prevented fault propagation through integrated buffer management and Rate Limiting features

Kakao completed an efficient migration by adopting a strategy of selectively internalizing only the features essential for actual operations, rather than reimplementing all of Fluentd's features.