Processing 41TB and 20 Billion Logs a Day in Real Time with ClickStack - The Hogwarts Library Project

The KakaoPay Securities DevOps team determined that the existing OpenSearch-based architecture could no longer handle the 41TB and over 20 billion log entries accumulating each day, and redesigned the entire pipeline from collection to storage and querying. The result was a new architecture centered on ClickStack (ClickHouse + OpenTelemetry + HyperDX), which reduced log ingestion latency from several minutes to under 20 seconds and lowered costs to around 14.4% of the previous level.

The core approach was to make the front end lighter, simplify intermediate processing, and restructure storage to fit log analysis needs. By switching Filebeat → OpenTelemetry, the team applied OTLP Proto batch transmission, improving Kafka transfer efficiency; compared to sending messages one at a time, batch transmission of 150 messages boosted throughput from 16.5 MB/s → 300 MB/s. Kafka CPU usage also decreased by more than half.

The Kafka structure consolidated Topics that had been scattered across services from over 300 down to 18, reorganizing logs by type such as std / nginx / transaction. Partitions were set up to a maximum of 150, with the number of consumers matched to common divisors to flexibly handle peak hours. Because of the nature of the securities business, traffic surges at 09:00 market open and 23:30 overseas market open (22:30 during daylight saving time), so the team chose to sacrifice some off-peak resources in exchange for headroom to respond to incidents.

The processor was also switched from Fluentd → OpenTelemetry Collector, introducing a tiered pool structure.

• Fast: core events and errors, official target within 2 minutes
• Common: general operational logs, official target within 15 minutes
• Debug: detailed logs for debugging, best-effort

This separation allowed reducing Debug resources during peak times while scaling up Fast, and reduced the over 1,000 Pods that had been running separately per service down to roughly 150. In practice, logs are now ingested into ClickHouse within 20 seconds of being generated, and per-core throughput improved from Fluentd's 150 records/sec to the OpenTelemetry Collector's 4,000 records/sec.

For storage, the team chose ClickHouse instead of OpenSearch. Since over 90% of query patterns were time range + field condition searches rather than full-text search, ClickHouse's columnar storage, ZSTD compression, and vectorized execution proved strong for large-scale aggregation and columnar queries. As a result, storage efficiency improved by roughly 90%, and overall costs were reduced by 78%.

For long-term retention, the in-house archiving tool ssak3 was configured to export ClickHouse data from the IDC to Amazon S3 in a one-way flow, which is then queried by ClickHouse on AWS. For custom logs, the team leveraged OpenTelemetry's Routing Connector and logtype extraction to process over 43 types of logs through a single pipeline, so that new log types could be added simply by modifying the Config. HyperDX was chosen as the query UI, enabling developers to explore logs in real time.