The Evolution of Dataflow for Large-Scale Machine Learning Data Processing

Since developing MapReduce 20 years ago, Google has continuously evolved its data processing platforms to support cutting-edge AI models like Gemini and Waymo. The resulting innovations from Flume are now applied to Dataflow, a fully managed batch and streaming platform, maximizing scalability, efficiency, and developer experience.

Key features introduced to solve large-scale scalability challenges include:

• Liquid sharding: Dynamically splits units of work during execution to resolve data skew and improve worker efficiency.
• Global compute: Optimally places workloads across global infrastructure by considering data locality and resource availability.
• Automatic pipeline optimization: Fuses consecutive operations into a single stage to reduce I/O and transition overhead.
• Rate-limiting external API calls: Controls the rate of external API calls to prevent overload in modern ML pipelines such as model evaluation.
• Tandem pools: Efficiently hosts and autoscales external model servers to support serverless remote inference.

It also offers optimization features to boost utilization of accelerators like TPUs:

• Heterogeneous worker pools: Allocates customized resources, such as CPU or TPU, for each pipeline stage.
• TPU-aware autoscaling: Prevents excessive initial allocation of TPU workers and supports efficient autoscaling.
• Duty-cycle policy enforcement: Automatically scales down TPU workloads when accelerator utilization is low.
• TPU fungibility: Schedules jobs to the most suitable TPU version and location based on quota and resource conditions.