Kakao Details Flink CDC Architecture for Real-Time MySQL Sync

Kakao's Data Analytics Platform team shared their production experience building a real-time data synchronization pipeline using Apache Flink and Flink CDC. The primary goal was to minimize load on the production database by offloading Change Data Capture (CDC) tasks to a separate system, enabling daily metric calculations without impacting core services.

Flink CDC vs. Debezium

While Debezium is a common CDC tool, Kakao found it limited when integrated with Kafka Connect. Processing tens of millions of records during the Snapshot phase could take days, often exceeding the default 3-day binary log retention period. This forced a complex separation of tools for snapshotting and streaming.

Flink CDC addresses these issues through distributed processing and Incremental Snapshot support. By dividing tables into chunks, it parallelizes the initial data copy, significantly reducing source database load. Additionally, Flink's Checkpoint mechanism allows for automatic recovery from failures during the snapshot phase, eliminating the need to restart the entire job.

Handling Snapshot and Binlog Consistency

The pipeline operates in two stages: an initial Snapshot followed by a Binlog Stream. To prevent data loss or inconsistency during the transition, the connector stores the GTIDs (Global Transaction Identifiers) of the first chunk in memory. The JobManager then identifies the lowest GTID from completed snapshot splits to determine the exact starting point for the binlog stream.

Chunk sizing is dynamic, calculated using a Distribution Factor based on the range of the chunk key column. For unevenly distributed keys, the system falls back to sequential queries, which limits parallelism and efficiency.

Customizations for Production Stability

Kakao implemented several custom modifications to the flink-connector-mysql-cdc to handle production edge cases:

• Stage Change Notification: Since Kafka topics cannot be changed dynamically in a running job, the system sends alerts via a Watchtower notification system when transitioning from Snapshot to Binlog Stream, including the relevant GTIDs.
• DDL Event Handling: To prevent sensitive information exposure and maintain exactly-once semantics, DDL events are skipped rather than stopping the job. The GTIDs of skipped DDLs are sent for manual verification and resynchronization.
• Database Switching Protection: To avoid connecting to the primary database, the system monitors for failovers. In a two-server setup, it checks DNS IP changes every 1000ms. In larger InnoDB Clusters, it queries the read_only setting to detect if it has accidentally connected to a primary node, throwing an exception and reconnecting to a secondary domain if necessary.

Version Recommendations

The article references Apache Flink 1.17.1 and Flink CDC 2.4.1. However, Kakao recommends using Flink CDC 3.x (specifically 3.1 or higher) for its improved snapshot logic and official support for MySQL-to-Kafka pipelines. Note that Flink CDC 3.x is currently compatible only with Apache Flink 1.18.