Kakao Reveals Spark Job Optimization Strategies for CDC Consistency Checks

Kakao's Data Analytics Platform Team revealed optimization strategies for Spark jobs performing consistency checks on CDC pipelines. To process over 300 tables and up to hundreds of millions of records daily, they balanced JDBC connection load and parallel processing levels by dividing MySQL operations into Fullscan, Keybased, and Limit modes. Notably, in Keybased mode, they improved query efficiency by extracting only changed primary keys and compressed partitions using coalesce. For Iceberg, considering the full-scan nature due to the absence of indexes, they periodically performed Compaction and snapshot expiration tasks. Additionally, they actively utilized Cache to prevent redundant operations caused by Lazy Evaluation, and applied environment-specific fine-tuning such as running in local mode and disabling broadcast joins in restricted environments like security zones.