Banksalad Introduces 'datapipe' to Let Data Analysts Directly Develop and Operate Pipelines

Banksalad developed and launched internally datapipe, an internal tool that allows data users to directly define, deploy, and manage pipelines, in order to resolve the bottleneck arising from the existing data engineer-centric pipeline operation structure.

Core Architecture and Features

• Abstracted Development Environment: Data users can focus solely on data processing logic, such as table specifications, dependencies, and loading logic. Complex infrastructure code such as EMR cluster configuration, Airflow DAG settings, and MySQL driver arguments is abstracted and hidden by datapipe.
• Self-Deployment and Permission Management: Instead of updating Kubernetes resources, it is implemented by syncing DAGs to the Airflow Scheduler, allowing users to deploy directly. Through the Airflow Web UI integrated with Fusion Auth, per-user DAG manipulation permissions are finely controlled.
• Automated Test Environment: When a PR is created, a test server (Scheduler, Web Server, Namespace) identical to the production environment is automatically deployed. Data output is isolated into a separate Namespace to prevent contamination of production data, and consistency is verified alongside pytest-based unit tests.
• Data Quality Management: By setting fail_conditions (load failure when criteria are not met) and dq_metrics (report generation) attributes in table metadata, data defects or duplication are blocked in advance. The configured quality metrics are reported to Slack daily.

Results and Implications

• After the introduction of datapipe, cases of pipeline development and operation by non-data-engineer roles (such as data analysts on product teams) reached 600 cases annually, an increase of more than 6x compared to before.
• Data engineers can focus on core pipeline and infrastructure issues, while product teams directly operate their own metrics and data products, improving development speed and flexibility.
• Future tasks include building simpler pipelines without direct infrastructure management and establishing ways to reduce infrastructure costs.