Kakao Shares Operational Case Study of 'Genesis', a Kafka Connect-Based ETL Platform, and Open-Sources Management Tool

Kakao's Ad Recommendation Team developed 'Genesis', a data platform based on Kafka Connect, to address issues such as lack of ownership, monitoring difficulties, and deployment inefficiencies in their existing Logstash-based pipelines. Genesis aims to provide ownership assignment, individual and overall throughput monitoring, web-based deployment, and data lineage management. It consists of a shared Kafka cluster, purpose-specific Connect clusters, JMX exporter/Burrow, and a Vue.js-based management web interface (genesis-web). During operations, the team open-sourced a web tool they developed for pipeline management on GitHub under the name 'kafka-connect-web'. Additionally, they emphasized the importance of cluster separation and provided a checklist for developing custom connectors (e.g., checking for open-source availability, Kafka topic input/output, and stateless processing). Future plans include advancing data discovery, state-based processing, SQL-based processing, and enhanced data lineage.