Iceberg Log Loading Strategy: DB Uses UPSERT, Server Logs Test APPEND with zstd Level 9 Reducing File Size by 51.4%

The Data Analytics Platform organization differentiated its Apache Iceberg table loading and operational strategies based on log types. DB logs are currently in production using UPSERT mode via Flink CDC, while server logs were tested using APPEND mode via Kafka and Flink.

Loading Method and Partitioning

DB logs use UPSERT based on Primary Key to reflect the latest data, with Bucket transform optimizing pruning. The commit cycle is set to 10 minutes, matching the Flink Checkpoint interval, to minimize small files.

Server logs involve massive data volumes, making the optimization cost of UPSERT greater than the benefits of deduplication, so APPEND mode was chosen. Deduplication is performed during the Spark metric calculation stage. Partitioning applies Identity transform, but Process time is stored as a string type to resolve timezone offset issues between UTC and KST.

Compression Performance and Storage Efficiency

Parquet format and zstd compression are used by default. In tests with an average of 3 billion server logs per day, applying zstd level 9 reduced file size by 51.4% compared to the existing ORC format. While increasing the compression level raises CPU usage, the impact on Kafka Lag and Flink processing delays was confirmed to be negligible.

Optimization and Monitoring

Compaction is performed using the binpack strategy, with the target file size set to 256 MB, matching the HDFS block size. For server logs, Compaction runs hourly on previous partitions, merging small files of approximately 6 MB.

Monitoring utilizes Prometheus, Grafana, and the internal TSCoke. Prometheus Pushgateway was introduced to collect batch Job metrics, preventing metric overwrites via honor_labels: true configuration and URL path-based grouping. Dashboards track changes in the number of data files, delete files, and manifest files to monitor for increases in small files.