Kakao's Development Journey of 'Pojang24': Automating Large-Scale Data Migration and Archiving

Kakao developed 'Pojang24', a system that automates large-scale data migration and archiving to transition from a commercial Hadoop platform to its proprietary Kakao Hadoop Platform (KHP). The legacy system imposed a heavy operational burden, requiring Hadoop administrators to manually write scripts and handle reprocessing upon failure, while lacking flexibility for schema changes and historical data processing.

Core Features and Architecture of Pojang24

Pojang24 is designed to allow users to manage data flows directly, adopting Apache Airflow as the workflow engine. Airflow supports scale-out, failover, automatic restart of failed tasks, and Catchup functionality for historical data processing, overcoming the limitations of the previous Oozie-based system.

• Data Migration: Provides a Base Docker image supporting HDFS Distcp and Hive queries, enabling users without Hadoop expertise to easily perform migrations.
• Hive Table Migration: Handles migrations flexibly for partitions with changed schemas by utilizing temporary Parquet tables.
• Kafka Ingestion: Allows simple configuration of data pipelines from Kafka topics to HDFS using Apache Flume.

Lessons Learned and Achievements During Development

Pojang24 was developed by 3 developers over 6 months, resulting in the creation of approximately 4,000 Flows and the migration of 40 petabytes of data. During development, performance issues with the Ruby-based Hive Connector and maintenance difficulties caused by excessive inheritance were identified, leading to the realization of the importance of composition-based design and connection management. The system significantly improved operational efficiency by providing visual monitoring of archiving status and sending alerts to users upon failure.