Case Study: Building kt cloud's Deck, an Operational Data Platform Unifying Scattered Data

kt cloud built the data platform Deck to unify operational data scattered across systems—such as logs, metrics, events, and usage—under a single standard. The goal is to reduce the inefficiency of repeatedly collecting and processing data, enabling operations, development, and business teams to analyze and make decisions based on the same data.

Deck adopts a Data Lakehouse architecture to handle structured, semi-structured, and unstructured data while ensuring the reliability required for billing and operational analytics. It combines the flexibility of a data lake with ACID transactions and schema management, applying two principles: building on pure open source and using Object Storage as the single source of truth (SSOT).

The platform is structured into layers by purpose.

• Collection: Collects and preprocesses various operational data reliably using Kafka and Vector.
• Storage: Stores raw data only once in Object Storage to reduce data duplication and management complexity.
• Query: Uses Trino to query Object Storage directly via SQL without loading data separately.
• Processing: Airflow manages the schedule, dependencies, and reprocessing of batch pipelines, while Spark handles large-scale cleansing and aggregation.
• Serving: StarRocks provides a low-latency OLAP environment supporting real-time Upsert and fast joins.
• Presentation: Configures an open-source BI environment with Superset to reduce license costs and vendor lock-in.

Each layer was designed independently to avoid bottlenecks that occur when collection through serving is concentrated in a single engine. By using a common storage base, the platform allows each layer to scale independently as services and data volumes grow. The billing pipeline is presented as a representative use case.