Avoiding Airflow Task Failure Alert Email Bombs

G Place Data Development Team operates a Data Lakehouse that loads data from various services into HDFS and connects it to query and BI systems. In this process, Airflow is used broadly, from ETL, consistency validation, backup, and Hive External table management to Github integration automation and sample data generation.

The problem occurred when multiple Tasks failed at the same time due to the same cause. Because Airflow's default notifications are sent per Task, when Tasks created in parallel failed in a chain reaction from the same incident, dozens of emails arrived at the administrator all at once, making it easy to miss the truly important alerts.

The solution was simple. All Tasks within the DAG were changed to not send emails even on failure, and the structure was reorganized so that only a single dedicated failure alert Task would gather the Tasks that failed at that point and send a single consolidated email.

The key points of the implementation are as follows.

• Keep email_on_failure in default_args set to True as is, but programmatically set email_on_failure=False for all Tasks within the DAG.
• Set only the Alert Task, which runs last, to email_on_failure=True, and configure it with trigger_rule='all_done' so that it always runs regardless of the results of preceding Tasks.
• Find the last Tasks using dag.leaves and connect the Alert Task downstream from them, while avoiding the issue of the Alert Task itself being included in dag.leaves and referencing itself, by using a separate list.

The Alert Task retrieves Task Instances using dag_run.get_task_instances() and filters only those with FAILED status. For each failed Task, it includes the task_id and log_url in the email, reads base_log_folder and log_filename_template from the logging section of conf to construct the actual log file path, and then appends the log content.

The issue of logs becoming too long was resolved using HTML's details tag. This allows the logs of failed Tasks to be collapsed and expanded within the email body, making it possible to identify the cause even within a single email.

Ultimately, by simply passing the DAG to remove_duplicate_alert(dag), you can receive one consolidated notification per DAG run instead of receiving numerous failure emails as before. This approach reduces the noise that was previously generated repeatedly from the same incident, while still allowing immediate identification of the location and logs of failed Tasks.