Toss Migrates DW Pipeline to 'Newtown'… Reduces DAG Code by 96% with dbt Adoption

Structural Reorganization and Strangler Fig Pattern Application

The Toss Data Warehouse team built a new platform called 'Newtown' based on domain-specific clusters and dbt to resolve bottlenecks and management difficulties in the central DW Airflow. By adopting the Strangler Fig approach, which involves running the new system in parallel with the existing one for validation, they achieved a stable transition without interrupting existing batch jobs during migration.

Code Simplification and Dependency Automation via dbt Adoption

By moving transformation logic from Impala queries to dbt models, dependencies were automatically managed using Cosmos. As a result, the DAG code for the advertising domain was significantly reduced from 2,794 lines to 100 lines, and manual dependency connection code (>>) decreased from 180 lines to 0 lines. Pipeline execution time was also shortened from 4 hours to 2 hours and 30 minutes.

Building AI Agent-Friendly Context

Emphasizing the importance of structured context (skills), metric definitions were added to model yml files and a semantic layer was applied. According to team hackathon results, including metric definitions in the knowledge base increased table selection accuracy from 25% to 75%, and when the semantic layer was used to generate and execute SQL via MetricFlow, the accuracy rate for calculation questions reached 82%.