Netflix Details Workload Attestation Design for Spark on Amazon EMR

Netflix engineers describe how to bridge the gap between cloud-provider identities and internal service identities for Apache Spark workloads running on Amazon EMR. Since managed compute hands processes only an AWS execution role, the team designed a workload attestation flow that exchanges this cloud identity for a first-class internal Metatron X.509 certificate.

The Identity Mapping Strategy

The design relies on a strict 1:1 mapping between each internal Data Project and a dedicated AWS IAM role. Because a single AWS account cannot hold tens of thousands of roles, Netflix shards these data project roles deterministically across a small pool of dedicated accounts. This mapping allows the platform to translate a provider statement ("this process is running as role R") into an internal statement ("this process is workload W").

The Attestation Workflow

The process involves five components: the control plane, the Data Project service, the Identity service, a Spark plugin, and AWS STS. The workflow proceeds in three steps:

1. Signed Claim: The control plane signs a metadata payload containing the application identity and mapped role, passing it as job configuration.
2. Proof of Possession: The Spark driver uses its AWS credentials to generate a pre-signed URL for sts:GetCallerIdentity. This URL acts as a transferable proof that the process holds the specific role's credentials, verified directly by AWS.
3. Corroboration: The Identity service fetches the pre-signed URL to confirm the role with AWS and verifies the control plane's signature. It issues a certificate only if the AWS-reported role matches the role specified in the signed metadata.

Handling Executor Scale

A Spark application can spawn thousands of executors, creating an identity amplification problem. Netflix chose to have executors inherit credentials from the driver rather than attesting independently. The driver attests once and distributes credentials over Spark's internal RPC, which is secured with authentication and AES-GCM encryption. This approach prevents the identity service from being overwhelmed by burst traffic, which could otherwise trigger rate limits or look like an attack.

Lifecycle and General Principles

Certificates are short-lived, so the driver JVM runs a timer to re-attest before expiry. Executors do not refresh; if they outlive their credentials, they exit and are replaced. The authors emphasize that attestation must be a repeatable operation, not just a bootstrap step, to prevent long-running jobs from failing. The core lesson is to anchor on one exchangeable primitive, require two independent claims (provider and control plane), and keep the signer scarce to maintain a clear trust story.