Kurly Adopts BigQuery to Resolve Data Pipeline Latency and Complexity

Kurly determined that timely data supply is critical for fact- and data-based decision-making, and adopted BigQuery. The existing Data Warehouse had four major problems: long latency, insufficient storage, slow query response time, and a complex loading process.

The existing architecture sent CDC logs from Oracle and Aurora DB to Kafka via AWS DMS, stored them in Amazon S3, and then loaded them into the warehouse using Airflow scripts. This process made it difficult to identify the root cause of bottlenecks that occurred from the source DB all the way to loading.

Key Points of BigQuery Adoption

To resolve these issues, Kurly applied BigQuery using the following strategies.

• Latency improvement: Removed the I/O bottleneck of the existing method of storing to S3 before running scripts, and used the BigQuery Streaming API to load data directly from Kafka to BigQuery.
• Storage management: Leveraged BigQuery's usage-based billing structure, while specifying TTL (Time To Live) per table partition to prevent excessive costs.
• Resource isolation: Separated the project for the data pipeline from the project for data querying to reduce resource contention and improve query response time.
• Pipeline simplification: Eliminated the complex intermediate storage step and switched to a structure that loads directly from Kafka to BigQuery.

Cost Management Strategy

Since BigQuery charges based on storage usage and data scan volume, cost management is essential. Kurly partitioned data by creation date to limit scan range and reduce costs. In addition, they purchased reserved slots for one project to handle large-volume queries, while separating out the remaining projects to be used without restrictions, aiming for efficient resource allocation.