Banksalad Migrates Spark Computing Environment from EMR to Kubernetes… Cuts Costs by 45%

The Banksalad Data Platform team is running a project to migrate its existing Amazon EMR-based Spark data analytics environment to Self-hosted Kubernetes. As operational difficulties persisted during EMR operation—including Autoscaling failures due to instance shortages, Driver termination, and complex configuration—the DevOps team determined that a Kubernetes-based transition, which they were already familiar with, would be advantageous in the long run.

Architecture Changes and Technical Challenges

In moving from the existing YARN-based execution structure to Kubernetes, the team leveraged the Kubernetes GA feature supported since Spark 3.1. Spark jobs are submitted via Airflow's KubernetesPodOperator, and awslabs' Hive Metastore client was included in the Spark image to reference the Glue Catalog. In particular, to resolve the issue of accessing Glue table s3:// protocol paths from outside EMR, they used forked code that applies s3a:// conversion logic.

Performance Optimization and Cost Savings

Unlike EMR's EMRFS, the Kubernetes environment can experience file commit delays when accessing S3, so the team applied the Magic Committer option to improve the performance of INSERT OVERWRITE queries. They also configured a Kubernetes Autoscaler using Spot instances to improve resource efficiency. As a result, they achieved about a 45% reduction in operating costs compared to before.

Remaining Tasks and Future Plans

The project is currently about 40–50% complete. Remaining tasks include collecting detailed metrics on Spark resource usage within Pods, building a Spark log management system, and resolving Glue/Hive dependency issues that may arise when upgrading the Spark version (from 3.0.3 to 3.1 or higher). Through these technical challenges, Banksalad plans to strengthen the stability and scalability of its data platform.