Kakao Data Services Releases 'AirZone', a PR-Based Personalized Airflow Testing Environment

Kakao's Data Services organization has released 'AirZone', a personalized Airflow testing environment designed to resolve inefficiencies in existing pipeline testing processes. Previously, complex preparation steps were required, such as setting up local environments, waiting for development Airflow synchronization, and accessing test containers, while interference issues also arose from sharing production resources.

AirZone uses GitHub PRs as the user entry point, automatically deploying an independent Airflow environment when a PR is created. Users can modify DAGs and test them immediately via browser-based IDEs (Jupyter, VS Code for Web) by simply clicking a link, without needing knowledge of Kubernetes or Helm.

Core Design and Architecture

• PR-Level Isolation: The repository name and PR number are designated as the Kubernetes namespace, fully separating the Airflow web server, scheduler, PostgreSQL, Jupyter, and other components. This ensures no impact on scheduling or resource usage of other projects.
• Asynchronous Deployment Structure: airzone-api only validates requests and responds immediately, while the actual deployment is handled by a Kubernetes Job executing the Helm chart. In case of failure, root cause analysis is facilitated through Job logs.
• Hadoop Authentication Integration: For Kerberos authentication, users' personal keytabs are injected or a shared keytab is used as a fallback, allowing Hadoop data access in the same manner as production.

Operations and Improvements

During the Beta operation period, features were enhanced before the official release based on collected feedback. To resolve delays in applying changes caused by checkpoint file issues in the initial Jupyter editor, support for VS Code for Web and local IDE connections was added. Additionally, environments left idle for more than 14 days are automatically deleted via CronJob to prevent resource waste. Since adoption, crews have been able to secure testing environments within approximately 5 minutes without infrastructure configuration, and a guide for setting up always-on personal Airflow environments is also provided.