Using Debezium and MSK Connect to Build a Product Data Pipeline

The Product Squad adopted Debezium CDC to solve database migration and synchronization issues while transitioning their existing monolithic system to MSA. The main goals were flexible synchronization between various databases, configuring a data flow pipeline, and efficiently tracking data change history.

Debezium is an open-source CDC tool that captures data changes in real time by leveraging DB transaction logs. The Product Squad adopted an architecture that configures a Debezium Source Connector on AWS MSK Connect and builds a separate Consumer server to utilize the processed data.

For DB configuration, the Logminer method was used for Oracle. The Xstream method can be advantageous in terms of performance, but it has high licensing costs and the constraint that library uploads are not possible on managed services such as MSK Connect. For MySQL, CDC is implemented by enabling binlog.