From 45-Minute Batches to Near Real-Time: A Migration That Unified Multi-Domain Data with Kafka

The benefit flags shown on product cards had to combine data from different domains—products, coupons, gifts, and promotions—into a single screen. Previously, a procedure accumulated over more than 10 years carried this logic, and reflecting a change could take up to 45 minutes, with tight coupling, validation difficulties, and large-scale recalculation issues all piling up together.

The nature of the data also differed by domain. Some domains could already be received as Kafka-based events, but some fields—like coupons or gift promotions—required coordination, and forcing all data to switch to real-time unconditionally wasn't a good fit either. So the team adopted a hybrid architecture that separated areas that could move to real-time from areas that still needed batch publishing.

The key point was that batch and streaming could update the same table simultaneously. Since simple flag polling had limits in a multi-instance environment, the team built a structure that broadcasts batch start/end signals via Redis Pub/Sub, while actual processing-completion status is checked through a separate flag. However, since Pub/Sub carries a risk of message loss, the system was designed to periodically re-check whether all Tasks had stopped, and to halt the batch if it wasn't safe.

Order-of-processing dependencies between domains were resolved using an Aggregation Topic.

• Rather than processing messages from each topic immediately, they are gathered into a single topic to coordinate ordering
• The product ID is used as the partitioning key, so events for the same product are routed to the same partition
• When there's no product row yet, coupon/event messages that arrive first are skipped, but at processing time the latest information is fetched via API and loaded

Idempotency was also prepared to guard against duplicate delivery and retries. Messages are received in batches, then go through deduplication and filtering by the latest publish timestamp, and the final write is handled via UPSERT, so that consistency isn't broken even if duplicates remain.

Just before the cutover, a Shadow Table was used to compare the results of the existing procedure against the new pipeline item by item. The two ran in parallel for about 3 weeks, and stability was verified even during high-traffic periods such as the Olive Young Sale, before switching over to the actual table. A rollback scenario was also prepared in advance so the system could immediately revert to the old structure if problems arose.

After the migration, the delay in reflecting benefit flags dropped to within a few seconds, and per-domain monitoring also became possible. Performance metrics improved significantly as well: buffer_gets dropped from 111.38M → 3.07M, a 97.2% reduction, and disk_reads dropped from about 120K → 8.48K, a 93% reduction. The display system no longer computes all the logic itself; instead, it now receives results that already reflect each domain's latest policy and combines them.