Containerizing the Airflow Development Environment with Docker Compose

To reduce the configuration mismatches caused by differences in how the Airflow development environment and deployment environment run, the local environment was rebuilt on a Docker Compose basis.

Previously, the local Python virtual environment and the production environment diverged, and OS differences also caused problems, making it hard to avoid situations where "it works locally but breaks after deployment." Containerization made it possible to test under the same configuration and to verify requirements.txt changes in an isolated environment.

The key to the setup was understanding Airflow's roles in a separated way.

• Scheduler: schedules DAG execution and records state to the metadata DB
• Executor: determines how tasks are executed and coordinates Workers
• Webserver: checks DAG status via the UI

The execution method differed depending on the Executor. The development environment used the simple SequentialExecutor, while the deployment environment used the highly scalable CeleryExecutor. To create an execution flow locally similar to the deployment environment, Redis, PostgreSQL, airflow-init, Scheduler, Webserver, etc. were bundled together with Docker Compose.

Compose was chosen because it's good at managing multi-container dependencies at once. The order was set so that Redis and PostgreSQL start first, then airflow-init initializes variables and connections, and finally the Airflow services run. Dependencies were defined with depends_on, and YAML duplication was reduced using fragment and extension syntax.

To ensure consistency with the deployment environment, the following were also incorporated:

• Using the Docker image used in production as the base image
• Injecting country, hadoop platform, phase, etc. as environment variables
• Reflecting DB connection, logging, and Python path settings
• Connecting the local DAG directory via volume mount

Once this work was done, a single command, docker compose up, could run Airflow locally in Celery executor mode.

The next task was handling this containerized environment for development and debugging in PyCharm. However, PyCharm's Docker Compose support had limitations, and it was also difficult to trace the cause of execution errors. So instead of a design where the image was a variable, YAML files were separated per country and changed to point to the latest tag, making them interpretable by the IDE.

For the development executor, a compromise was made to use LocalExecutor instead of the resource-heavy CeleryExecutor. It was lighter and faster while still providing more parallelism than the existing SequentialExecutor. The structure was organized by separating common settings into a shared YAML and managing Local/Celery-specific YAML files separately.

Run/debug configurations were also redesigned for running and debugging in PyCharm. The approach of separating initialization from DAG execution was convenient for reuse but carried a heavy management burden, while unifying everything under a single docker compose command required re-running initialization every time. In the end, the latter was adopted as the default, using the --force-recreate and --exit-code-from options to automate test termination and container cleanup.

Two results were ultimately obtained.

• Being able to check DAG execution locally in a way close to the deployment environment
• Being able to set breakpoints and test/debug DAGs in PyCharm

In addition, Makefile targets were used to unify test and Airflow-related commands, and management costs were reduced by centering the local airflow.cfg and the deployment environment's Helm values around Helm value. By also organizing the Run configuration naming convention, a development environment that remains easy to maintain even as the number of DAGs grows was created.