Data Projects: Managing Data Assets at Netflix Scale

Netflix's data platform is massive, encompassing millions of tables and tens of thousands of scheduled workloads. Previously, the approach was to set an Access Control List (ACL) for each individual asset and run workloads under the ID of the individual who wrote them.

However, this approach had clear management limitations: permission updates were difficult during organizational restructuring, and workloads would break when the responsible person left the company or moved to another team. In particular, as permission management became more complex, this led to the problem of granting overly broad permissions for the sake of security.

To solve this, Data Projects was introduced, providing two core capabilities.

• Logical grouping of assets: Related assets such as tables and workflows are bundled together into a single container called a Project for unified management.
• Sustainable synthetic identity: A Synthetic Identity that does not depend on any specific individual's lifecycle is assigned, ensuring that workloads keep running stably even when the responsible person changes.

Now, instead of individually modifying permissions for hundreds of tables, administrators only need to update permissions for the single Project that contains those tables. Each project supports efficient permission management by granting roles such as Contributor or Viewer to targets like users, groups, and applications.