Spark Connect on Kubernetes #1: Building a Robust Spark Connect

The Toss Securities Data Infra team operates a Kubernetes-based Spark Connect service so that analysts and engineers can use Spark without complex configuration. The first topic in this series covers building a 'robust Spark Connect' that maintains stability even in an environment shared by multiple users.

Whereas the existing Classic Spark had a structure in which a new Driver was created for each application, Spark Connect adopts a model in which the Driver is separated out as a pre-running server, and clients request computations via gRPC. This is similar to querying a database, and it reduces the burden of installing heavy libraries on the client side.

However, this Long-running server model conflicts with Spark's existing '1 app = 1 workload' design principle, causing the following issues.

• Single Point of Failure: Since all sessions share a single Driver JVM, if a specific user's query exceeds the Executor failure threshold, the entire server shuts down and all sessions are interrupted.
• Resource Contention and Scheduling Limitations: While logical isolation between sessions is possible, physical resources such as CPU and memory are shared. In addition, the default FIFO scheduler has no preemption feature, so if a heavy job occupies the slots, other users' jobs fall into a waiting state.