Kakao Reveals Development Journey of Internal TSDB 'TSCoke'… Performance Optimized with ART and LSS

Kakao's Distributed Technology Cell developed the internal TSDB service TSCoke to resolve the complexity of existing monitoring metric management and provide a unified analysis environment. Initially, a prototype was built to store metrics on a distributed Key-value store Redicoke by utilizing leftover storage space, but in the actual production environment, it faced two serious issues: Out-of-memory and Timestamp inversion.

Maximizing Memory Efficiency

As the Index for Label search grew massive, causing memory shortages, Adaptive Radix Tree(ART) and Dictionary coding were introduced. By applying Dictionary coding to map string Labels to Integers, the Key distribution of ART was made denser, significantly improving memory efficiency. Tests with 100,000 real data points confirmed improvements in both memory efficiency and performance compared to the existing sync.Map.

Resolving I/O Bottlenecks

The Timestamp inversion phenomenon occurring in the Master-Master Replication structure caused intermediate insertions into compressed data, sharply increasing I/O load. To resolve this, instead of heavy Embedded KV Stores like Bolt or Badger, a Log-structured Segment Store(LSS) tailored to NAND I/O characteristics was implemented in-house. LSS increases IOPS by tens of times and significantly reduces memory consumption by chunking Slots with similar expiration times and writing them in an Append Only manner.

Service Status

TSCoke, which beta opened on June 9, 2022, based on Kakao's Kubernetes service DKOS, is actively operating as of August 4, 2022, storing over 300 million Series. Currently, distributed query functionality and Label extraction performance have been added, and future development plans include Prometheus Alarm, Recording rule, and external storage integration features.