SK Planet Develops 'Router', a Real-Time Derived Data Generation Platform Based on Kafka and Spark

SK Planet developed the real-time streaming platform 'Router' to solve the data processing latency problem that occurred in its existing Hadoop-based Data Integration Cluster (DIC). The existing approach followed a batch processing method in which derived data was generated in the early morning of the next day after data in Hive tables had been fully collected, but this caused significant delays in analysis and service delivery.

Router adopted a structure in which real-time data ingested via Kafka is immediately transformed and loaded through Spark Streaming. Data flow can be visualized in a Directed Graph form, allowing pipelines to be configured via the UI, and it was designed so that Hive Query logic used in existing batch jobs can be reused as-is, improving development efficiency.

Core Architecture and Features

• Data Flow Management: Data is collected via Connect based on Kafka Connect, and transformed via Processor based on Spark. Stream-Processor transforms Kafka data in real time, while Hive-Processor sends Hive table data to Kafka.
• Resource Optimization: Resources can be dynamically adjusted per pipeline in a Yarn environment. Allocation of resources suited to data volume is possible by setting the number of cores and memory for the Spark Executor in the UI.
• Support for Various Storage Systems: Data can be loaded into various storage systems such as Elasticsearch, S3, and HBase, and currently the primary method used is loading into HDFS followed by Hive queries.

Monitoring and Ensuring Stability

To ensure the stability of real-time processing, a 3-stage monitoring system utilizing Grafana and Burrow was built.

1. Spark Job Status Check: Tracks Yarn's applicationId to check whether the Job is running normally, and sets up automatic restarts when necessary.
2. Throughput Monitoring: Calculates the number of Kafka data records processed per micro-batch, logs it to Elasticsearch, and sends alerts if it falls below a threshold.
3. Data Delay (LAG) Detection: Calculates LAG, the difference between Kafka's last loaded offset and the consumer group offset, using Burrow to identify data consumption delays. Alerts are received immediately upon abnormal state changes such as WARNING or ERROR.

Through this system, SK Planet has made it possible to generate derived data simultaneously with data collection, greatly shortening the speed of deriving insights, and plans to expand it into a data platform for generative AI technology in the future.