Kakao Releases 'kafka-sink-connector' Open Source for Ad Data Pipelines

Kakao's Ad Recommendation Team has open-sourced kafka-sink-connector, a custom connector developed for ETL and real-time model training of ad log data. This connector is utilized within the Genesis data platform based on Kafka Connect, performing filtering, branching, and sampling of vast raw ad stream data according to placement-specific requirements.

The existing data pipeline utilized tools like Logstash, but operational efficiency was low in environments where data branching and filtering logic per placement changed frequently. kafka-sink-connector addresses these issues by leveraging the extension structure of Apache Kafka Connect, developed as a custom connector to replace Logstash's filtering capabilities.

The key features of kafka-sink-connector are as follows:

• Filtering and Branching: Filters data using jsonPath conditions and branches it to specific topics.
• Metadata Injection: Parses and injects message keys and timestamps.
• Co-partitioning Support: Injects message keys when co-partitioning with external topics is required, preparing for Kafka Streams joins.
• Mirroring and Sampling: Samples data at a specified rate or serves as a lightweight mirroring pipeline instead of the heavy MirrorMaker2.

This tool implements logic within the put() method of SinkTask and was released to help organizations using Kafka Connect simplify pipeline operations.