How Pinterest drastically reduced Apache Spark OOM errors

Pinterest introduced Auto Memory Retries, which automatically retries failed tasks on larger executors, to reduce OOM (out-of-memory) failures in its large-scale Apache Spark environment.

The cluster runs on Kubernetes and handles 90K+ Spark jobs per day with shuffle at a scale of hundreds of PB. Previously, it was difficult to blindly increase executor memory, and since memory requirements for stages and tasks vary by job, manual tuning alone had its limits.

The core idea is to separate the resource profile at the task level. When an OOM occurs, it first increases cpus per task to retry on the same default executor, and if that still fails, it spins up a new larger physical executor. At this point, when the base profile is registered, immutable retry profiles of 2x, 3x, 4x are also created together and used sequentially, and when off-heap memory is enabled, that is scaled up together as well.

To do this, Task, TaskSetManager, TaskSchedulerImpl, and ExecutorAllocationManager were extended.

• Task has an optional taskRpId indicating when it differs from the parent TaskSet.
• TaskSetManager automatically assigns the next retry profile when an OOM failure occurs, and manages task indices per profile.
• TaskSchedulerImpl also places tasks with increased CPU settings on the default executor, maximizing reuse of existing resources.
• ExecutorAllocationManager tracks the number of pending tasks per retry profile and spins up larger executors when needed.

At runtime, instead of using SparkListener directly, a Pinterest-specific subclass was created that overrides only the necessary functions, so that when the feature is off, existing Spark behavior is preserved, and the new logic only runs when it's turned on. The SparkUI also displays the task resource profile id, allowing operators to easily see the retry status.

The rollout expanded from ad hoc submission, going 0%→100%, and then scheduled jobs were expanded in the order Tier 3 → Tier 2 → Tier 1. The dashboard tracked the number of recovered jobs, saved MB-seconds, saved vcore-seconds, and the number of jobs that failed even after retry, and ultimately overall OOM failures were reduced by 96%.

The lessons learned were also clear. For very large TaskSets, the process of finding the active profile could become slow, so an additional index was created, and the timing of retry profile creation was adjusted to account for cases like Apache Gluten, where resource profiles are registered after the application starts, as well as cases where Scala/PySpark users register profiles directly.