Kakao Releases Open-Source 'ADT' for MySQL Data Transfer and Shard Redistribution

Kakao has released ADT (Almighty Data Transmitter), an open-source tool that allows users to collect MySQL data and transfer or process it to other databases. This tool combines CDC (Change Data Capture) and ETL capabilities and was developed to efficiently move data during shard redistribution or sharding rule changes.

Key Features and Use Cases

ADT collects existing data and real-time change data from MySQL via Table Crawler and Binary Log Receiver. The collected data is passed to a Custom Handler implemented by the user and can be utilized for various purposes.

• One-time Migration: Shard data redistribution, schema changes, daily data copying to OLAP DBs, etc.
• Real-time Migration: Read load balancing via data copying to NoSQL, sending push notifications upon detecting specific events, GPS-based shard reconfiguration, etc.

Application Cases and Future Plans

One of the application cases so far is the migration of a service where the table schema changed drastically, making it impossible to handle with ALTER TABLE. Using the initial version of ADT, zero-downtime changes were not possible, but the data migration was successfully completed. The custom handler for shard redistribution, which is the core goal, is currently undergoing stability and consistency verification with DB engineers. Currently, only MySQL is supported, but contributions from the open-source community are welcomed to expand support for other databases in the future.