Log Pipeline Improvement Story - Defining Problems in the Existing Pipeline and Applying Solutions

The existing log pipeline operated with the flow KDS → Firehose → S3 → Airflow classification → GCS → BigQuery. Since multiple log types were mixed in a single file, classification work was essential, and this process led to longer processing times and duplicate storage across S3 and GCS.

There were three major problems. First, all types had to be classified at once, which was inefficient, and a failure in the classification stage created a single point of failure that halted the entire loading process. Second, since it was Airflow-based batch processing, data freshness was low, which didn't meet real-time analysis requirements. Third, there was no schema change management system, so producers and consumers couldn't see the same rules, and there was a significant operational burden from manually changing the BigQuery schema.

The direction for improvement was to create a single source of truth (SSoT) and transform the batch-centric structure into a streaming structure. To achieve this, the messaging platform was switched to MSK, and instead of Firehose, a custom Kafka Consumer was directly implemented for consumption. This allows type-based classification and partitioning to be handled directly at the Consumer stage, eliminating the existing classification work and enabling near-real-time loading.

For schema management, Data Contract and Schema Registry were introduced. Protobuf was used as the common schema format, and Buf automated linting, formatting, compatibility verification, and code generation. The change flow was organized so that producers and consumers go through PR review and testing, and Buf plugins generate BigQuery schemas, Python classes, and Java classes, which are then deployed to each service.

In the new architecture, the server sends logs to MSK, and the Consumer reads them using ProtobufDeserializer, processes them into the necessary format, and stores them in GCS. GCS becomes the source for BigQuery External Table, and to compensate for the performance limitations of external tables, data is loaded into an internal table daily, and finally, the View Table serves both the external table and internal table together. As a result, data is provided faster and in a more consistent manner, and schema changes can now be centrally controlled.