In-Depth Analysis of Apache Iceberg and Flink CDC Integration: Performance Optimization and Operational Case Studies

The previous method of loading entire MySQL datasets into HDFS via daily batch jobs caused severe database load, including reaching 100% disk usage. To resolve this, the architecture was switched to integrate Apache Iceberg and Flink CDC, reflecting only real-time changes. This approach eliminated the full data loading stage and enabled Spark queries without database load by reflecting real-time data into Iceberg tables via CDC.

Flink CDC Direct Loading Architecture

The existing Debezium approach based on Kafka Connect suffered from reduced throughput due to heavy JSON-formatted messages. To improve this, a structure was adopted that loads data directly into Iceberg using the Flink RowData format, bypassing Kafka. This structure significantly improved throughput by lightening the message payload.

• Performance Comparison: The Kafka transmission structure averaged 5k msg/s per parallelism unit, whereas the direct Iceberg loading structure averaged 15k msg/s per parallelism unit.
• Maximum Throughput: With parallelism set to 20, throughputs of up to 360k msg/s and 280k msg/s were recorded, respectively. Database load testing confirmed scalability up to a maximum parallelism of 45.

Table Configuration and Partitioning Strategy

Iceberg tables use Format Version 2, with Parquet as the file format and zstd compression applied to enhance Trino query performance and GC stability. The write mode remained the default COW (Copy-on-Write), though analysis showed negligible practical impact due to the characteristics of the Flink loading logic.

Partitioning uses Bucket partitions based on the primary key, selecting columns with high cardinality in the case of composite keys. Modulo value testing revealed that Spark query times tend to increase from a value of 50 or higher, so the Modulo value was ultimately set to 5 to prevent excessive file fragmentation.

Scan Planning and Compaction Optimization

Iceberg's scan planning selects only the necessary data and delete files based on partition information, sequence numbers, and statistics. Specifically, Equality Delete files are applied only to past data files with smaller sequence numbers and are finalized only when the minimum/maximum value ranges of the equality columns overlap, optimizing performance.

Compaction is critical for maintaining query performance. At a scale of 3 billion records, without compaction, query time on day 7 spikes to 60 minutes, but with compaction, it takes 39 minutes before query time recovers to approximately 1.7 minutes. However, to resolve issues where delete files accumulate under certain conditions, features such as rewrite-all or rewrite_position_delete_files must be utilized.

Limitations of Integrated Operation for Sharded Tables

An experiment was conducted to integrate multiple sharded tables into a single Iceberg table, but application to production services was postponed. Integrating 32 tables (2.7 billion records) resulted in an average sourcing time of 270 seconds, offering a performance advantage over sourcing a single table 32 times (640 seconds). However, CommitFailedException occurred during concurrent commits from multiple Flink jobs. While retry settings could mitigate this, ensuring stability at a scale of 100 or more shards proved difficult, leading to the final decision not to adopt this approach.