Kurly shares strategy to prevent data loss when connecting DB data via Kafka Connect

Kurly's Data Service Development Team operates a pipeline that integrates various DB data into an analytics platform using Kafka Connect. The JDBC source connector can send RDBMS data to Kafka using only JSON configuration without separate coding, making it advantageous for building repetitive pipelines.

However, since the JDBC source connector is query-based, unlike log-based CDC (Change Data Capture), it cannot detect delete or update events, and data loss can occur due to data ordering issues caused by transaction delays. In particular, when there are long transactions or network delays, events that should have been loaded first can be missed.

To mitigate this issue, Kurly uses the timestamp.delay.interval.ms option. This option maintains data integrity by introducing a certain delay before loading into Kafka, excluding data from incomplete transactions. However, if the option value is set too large, loading delay increases, so it must be configured to balance loading delay and data loss according to data characteristics such as transaction timeout duration.