Running Apache Spark experiments in my sleep (and on a plane)

A Netflix engineer transformed a two-month debugging effort into a streamlined process by shifting from single-run optimization to parallel experimentation. The core challenge was that critical Spark failures, such as executor Out-Of-Memory (OOM) errors, only occurred at full production scale due to data skew, making small-sample testing ineffective. By leveraging Dataflow and Maestro to isolate experiments via Git branches, the engineer could run five concurrent jobs against unsampled production data, reducing total execution time to that of the single slowest job.

Operational Automation with AI Agents

To manage the complexity of parallel runs, the engineer deployed an AI agent to handle operational tasks rather than code generation. The agent managed scheduler configurations, deployments, and metric polling, allowing the engineer to query results like "Which run is closest to OOM-free?" via natural language. This workflow was packaged into a reusable skill (/sandbox-deploy-and-run) that handled the full lifecycle from preflight checks to auto-diagnostics, enabling the engineer to monitor experiments remotely over spotty Wi-Fi using tmux sessions on Netflix’s Workbench cloud environment.

Root Cause: Memory Pressure from Event Bag Structure

Investigation revealed that the root cause was not total dataset volume but the memory footprint of individual event bags aggregated by collect_list. Two primary factors inflated event weight:

• Data Type Overhead: Upgrading from a legacy architecture introduced Scala Option types, forcing Spark to wrap primitives in JVM objects, which increased garbage collection pressure.
• Bloated Attributes: Low-value string fields (e.g., referrer URLs) added significant weight per event, which multiplied across collect_list operations.

Unboxing Option types to plain primitives reduced peak executor memory usage significantly, while slimming event bags by hoisting session-invariant metadata further alleviated pressure.

Adaptive Query Execution Pitfalls

Two critical insights emerged regarding Spark’s Adaptive Query Execution (AQE):

1. Broadcast Join Accumulation: AQE rewrote sort-merge joins into broadcast hash joins. While individual joins stayed under the 400 MB limit, the cumulative pinned memory reached 3.1 GB per executor, causing OOMs.
2. Window Operator Buffering: A naive fix to truncate events after a Window operation failed because Spark buffers the entire partition before evaluating the window. The solution required capping events before the Window stage to prevent memory spikes.

Adjusting spark.sql.aqe.broadcastThreshold to 10 MB resolved the broadcast accumulation issue without disabling broadcast joins globally. The author emphasizes that while tuning buys time, the underlying data model of aggregating unbounded event streams into single rows has a hard scalability ceiling.