Data Engineering at Olive Young

Olive Young's data engineering aims to deliver data in a usable state to those who need it, covering a broad scope that goes beyond simple data pipeline development to include establishing data processes and building data platforms.

Previously, the company used a traditional IDC-based ETL system to load source DB data into a Data Warehouse (DW) and utilize it for BI and analytics work. However, a new design was needed to address rapidly increasing data intake and diverse data provisioning requirements.

Accordingly, Olive Young adopted the highly scalable GCP (Google Cloud Platform) to rebuild its data platform. It now uses Airflow and Spark to accommodate a variety of source data, and provides a powerful analytics environment leveraging BigQuery.

The company has also expanded the scope of its data engineering by extending into log collection, taking advantage of cloud benefits, and is pursuing further expansion toward a data service platform based on an MSA architecture.