PostgreSQL CDC Pipeline Simplified with Kafka Connect… ES Indexing Speed Improved by 47%

The data synchronization pipeline from PostgreSQL to Elasticsearch for a content monitoring system that has been in operation for 10 years has been completely restructured into a Kafka Connect CDC architecture. The previous method involved multiple stages, including queue tables, batch apps, and RabbitMQ, resulting in numerous failure points and difficulty in root cause analysis. The new structure achieves near-real-time synchronization by leveraging Debezium and the ES Sink Connector.

Architecture Changes and Tech Stack

The new pipeline is based on PostgreSQL's Logical Decoding. The Debezium Source Connector captures WAL changes and sends them to Kafka, where an intermediate Python app transforms the data according to business logic. Subsequently, the ES Sink Connector consumes Kafka messages and indexes them into Elasticsearch. Kafka Connect is deployed in a Kubernetes environment via Strimzi, ensuring stability in distributed mode.

Key Results and Improvements

Simplifying the pipeline eliminated unnecessary dependencies, making it easier to identify the causes of failures. In particular, intermittent data consistency issues have been completely resolved, and ES indexing speed has improved by up to 47% compared to the previous system. In the Debezium configuration, ByLogicalTableRouter is used to route events from multiple tables into a single topic, while the ES Sink Connector applies dynamic mapping using the schema.ignore option to enhance flexibility.