Deep Dive into Flink CDC to Apache Iceberg Integration: Handling Sharded Tables and Commit Conflicts

The team implemented a pipeline using Flink CDC to load MySQL change data directly into Apache Iceberg tables, bypassing Kafka Connect to reduce message overhead. They utilized UPSERT mode with format-version 2 and a 10-minute checkpoint interval to balance stability and real-time availability. To handle sharded MySQL tables, they consolidated multiple shards into single Iceberg tables using partitioning strategies that include an 'identity' column to distinguish shards and bucket partitioning for performance. Testing showed that consolidating 32 tables (2.7 billion rows) into one Iceberg table resulted in an average sourcing time of 270 seconds, compared to 20 seconds for a 90 million row table, indicating higher efficiency than running 32 separate tasks. The team faced commit conflicts when multiple Flink jobs wrote to the same table, encountering errors when the base metadata file changed. Increasing retry counts was insufficient for stability, so they avoided features that could stop jobs. Compaction was essential to manage small files; without it, query times for a 3 billion row table increased from 2.8 minutes to 60 minutes in a week, but compaction reduced it to 1.7 minutes.