Building a CDC Pipeline from Oracle to MongoDB

Kakaopay Securities pursued the construction of a CDC(Change Data Capture) pipeline centered on Kafka to efficiently utilize data distributed as a result of MSA expansion. The core of this project is building a pipeline that transfers data from Oracle to MongoDB for real-time analysis of user data.

The architecture was composed as follows.

• Oracle and the Kafka Cluster were placed on IDC physical servers.
• MongoDB was configured on a separate physical server for this build.
• Kafka Connect was built on a Kubernetes environment and managed with Helm to flexibly handle sudden load spikes.

Debezium was used as the data source connector, and the MongoDB Sink Connector was used as the sink connector. However, there was a technical constraint in that the CDC Handler of the MongoDB Sink Connector currently supports MySQL and PostgreSQL, but not Oracle.

To resolve this, they analyzed the source code of the Debezium CDC Handler and reviewed an approach to directly implement a custom handler for Oracle, similar to the existing supported DBMS handlers.