Kurly's Journey Building a Near-Real-Time Demand Forecasting Pipeline with Dataflow

Kurly's Data Platform Team shared their experience building a near-real-time demand forecasting system pipeline using Apache Beam and Google Cloud Dataflow. The demand forecasting model analyzes time-series correlations based on one year of historical data, and near-real-time processing was essential to reflect customer order data that changes in real time.

Background for Adopting Dataflow and Architecture

Kurly chose Dataflow, which supports large-scale distributed processing and streaming processing in the GCP environment and enables automatic scaling and resource optimization. Apache Beam is a programming model that can handle both batch and streaming pipelines, and it runs by integrating with Dataflow's backend performance. The data flow structure involves loading order data collected via AWS MSK (Kafka) into BigQuery, which Dataflow then processes.

Pipeline Execution Principles and Precautions

In Dataflow, during the preparation stage (queued state), the Launcher VM creates the graph, and during the execution stage (running state), the Worker VM performs the actual code. Since batch pipelines take time due to Launcher creation/termination each time, setting 1-minute batches for near-real-time performance can fail. On the other hand, streaming pipelines run continuously, so after the initial launch they operate stably in the same environment. However, since BigQuery is a bounded data warehouse, it behaves like batch mode when applied to a streaming pipeline.

Building Flex Templates and CI/CD

For repeated deployments, they used Flex Templates. Flex Templates increase reusability by storing Docker images and template spec files in Artifact Registry and GCS. They automated the CI/CD workflow via GitHub Actions, proceeding in the order of deleting existing streaming jobs, building and pushing Docker images, building Flex Templates, and running Dataflow jobs.

Operational Tips and Permission Management

For streaming pipelines, using Streaming Engine is recommended. This reduces the burden on Worker VMs and helps cut costs, but it may incur separate charges. Additionally, the save_main_session=True setting should be used to prevent issues with global variables and module references. For permission management, both the Dataflow Service Account and the Worker Service Account are required, and care must be taken since VPC communication errors, among other issues, can stem from insufficient roles on the Dataflow Service Account.