How Karrot Built DT Platform, an ELT Platform for Migrating 200+ DBs

Karrot transfers data from 200+ DBs across its global services in Korea, Japan, Canada, and more into its in-house data warehouse, BigQuery. Each region and environment (alpha, prod) has its own transfer cycle and configuration, resulting in a complex structure.

In the existing ELT approach, the pipeline's configuration and execution code were tightly coupled within a single repository. Every time a new table was added or a configuration changed, a PR (Pull Request) had to be created, which increased the operational burden on the Data Value Team and made it difficult to get an overview of pipeline status at a glance.

To solve this problem, Karrot's Data Value Team built an in-house ELT platform called DT (Data Transfer) Platform. By separating configuration from code, they improved operational efficiency and laid the groundwork to handle ELT/ETL pipeline management in the AI era going forward.