FAST: Data Pipelines, Now on the Web

Naver Pay's Intelligence Platform team consolidated its scattered batch methods into FAST for data utilization. FAST stands for FDC automated self tasker, a data pipeline creation tool that deploys to Airflow DAG just by entering input on the web.

Previously, methods varied depending on the purpose.

• Zeppelin Cron: Scheduling notebooks for periodic execution
• Airflow-based batch: Data engineers directly creating DAGs
• Redash Scheduled Query: Operating batches for dashboard updates

This structure placed a heavy burden on both users and operators. Non-developers had to overcome knowledge barriers such as YAML, git, jenkins, and github, and needed to communicate with data engineers every time they made modifications. Additionally, Redash was inconvenient for periodic mailing, and there were cases where batches failed due to heavy queries.

To reduce these problems, FAST adopted an intuitive web-based creation flow. Actual batches use Airflow for stability and maintainability, while the web interface allows easy definition of tasks such as Hive(JDBC), Bash, TextMailing, ScreenshotMailing, and Join Task. Users simply add the Tasks they need and fill in only the required values, and the entered pipeline is automatically converted into Python code and a YAML configuration file before being deployed to Airflow.

The core features were also organized to fit actual usage.

• Batch inquiry and batch creation handled on the web
• Query validation per Task to detect errors before execution
• Frequently used patterns provided as templates
• Ability to copy other users' batches, check deployment history, rollback, and batch duplication
• Enhanced collaboration convenience with team account (keytab) support

Before adoption, users had to handle everything themselves, from development environment setup to learning YAML, writing example code, and responding to deployment failures. After adoption, users only need to look at existing examples, enter input values, and save/deploy, reducing the required knowledge to roughly SQL and how to use FAST.

Through FAST, the team lowered the entry barrier for users and reduced the management and communication costs of scattered components. Going forward, they plan to add AutoML and pyspark support for non-developers, expanding beyond a batch tool into an All-in-one data literacy platform.