Toss Securities Releases Spark Tenant Operator for Managing Spark Connect Servers

The Data Infra team at Toss Securities developed the Spark Tenant Operator to manage the lifecycle of Spark Connect servers. While the existing Apache Spark Operator handles the execution and recovery of individual Drivers, the Tenant Operator groups multiple SparkApplications into a single tenant to manage server count and configuration changes.

Role Division Between the Two Operators and Blue-Green Replacement

The Tenant Operator creates and replaces SparkApplications according to user-declared configurations. During server replacement, it applies a Blue-Green deployment strategy, routing only new sessions to the new server while keeping the old server in a drain state until existing sessions terminate. This prevents interruption of running tasks.

Readiness Verification and DNS Issue Resolution

To verify the actual readiness of the Connect server, an exec probe attempting a gRPC connection was introduced. Additionally, the spark.kubernetes.executor.useDriverPodIP=true setting was applied to prevent deadlocks caused by logic waiting for the Ready state when Executors resolve the Driver's Service DNS. Furthermore, the spark.kubernetes.driver.service.publishNotReadyAddresses setting was used to separate Executor connections from readiness to accept Connect requests.

Resolving Memory Leaks in Long-Running Environments

Resolved an issue where heap usage remained stable but total memory increased, leading to termination via OOMKilled. By limiting the number of arena instances to reduce glibc memory allocation contention (MALLOC_ARENA_MAX=2), memory growth was prevented during 72-hour observations. In production environments, the value max(4, cores) was applied considering the number of cores.