Retrospective on Exhibition Deal Internalization Project: MongoDB-based Data Construction and API Improvement

By moving Exhibition Deal lookups from a denormalized table based on OracleDB to a collection based on MongoDB, both response latency under heavy traffic and operational complexity were reduced.

Previously, product information was queried from MongoDB and deal information from OracleDB separately and then combined, which meant every call required two DB connections, and whenever an issue occurred, deals and products had to be traced separately. The creation logic and update cycle of the Oracle Serving table also depended on another team, which made unit testing and immediate response difficult.

This migration was carried out as an extension of the THOR platform. THOR is a MongoDB-based data platform that the exhibition service development team directly builds and operates to hold only the information it needs, designed to reduce the limitations and maintenance costs of the PL/SQL-based Oracle Serving table.

Because the scope of impact was wide and the data changed organically, the migration work could not be completed as a simple substitution. There were about 60 APIs that used deal information, and since the actual fields used and the conversion logic differed slightly for each API, a validation process for comparing as-is vs to-be API responses was ultimately incorporated as well.

Data construction was composed of a combination of Spring Batch, Kafka, and MongoDB.

• The part that extracts the required deal information was separated from the creation logic of the Oracle Serving table to build a collection batch.
• A time-deal product information update batch was added to improve the accuracy of time deals.
• Deals with large amounts of data were split into separate batches, and collection speed was increased using thread-pool=5 multithreading.
• The extracted deal information was published to Kafka using the Zero Payload approach, passing only keys.
• At the loading stage, topics were separated by batch, and large-volume deals were consumed with concurrency=3 to quickly reduce lag.
• To reduce the burden during API lookups, processing logic was concentrated in the construction stage, and single and composite indexes were designed according to the fields being queried.

Validation was also split into two stages. Initially, the total counts, differences, and random samples between the new MongoDB collection and the existing Oracle table were compared, and later the scope was expanded to compare actual API response values using schedulers running at 5-minute, 10-minute, and 20-minute intervals. The comparison results were collected via Logstash and the ELK stack and checked on a dashboard.

There were also some trial and error during operation.

• When product information was placed as an embedded document under the deal, the sub-product information was only updated at the time the deal was updated, causing cases where the latest product information was not reflected. Eventually, the use of embedded product information was discontinued and the logic was simplified.
• As data unexpectedly surged from 160,000 to 370,000 records, the collection batch time grew to as much as 10 minutes, and during that time a problem occurred where deals were not displayed. This was resolved by strengthening multithreading and separating batches and topics.
• It was confirmed that making the rollback plan too granular actually increased confusion in emergency situations. Afterward, the rollback mechanisms and steps were minimized and simplified into a 1-step rollback-centered approach.
• Initial validation only targeted the loaded data, which could miss API-level issues, and it was ultimately concluded that comparing response results was more important.

As a result, removing the Oracle connection significantly improved response latency under heavy traffic, and the deal update speed also became faster, allowing more stable handling of data surges during event periods. The debugging point was also simplified from a structure that went back and forth between Oracle and MongoDB to being centered on MongoDB.

Going forward, there are plans to more clearly separate fields that do not need updates after initial loading from fields that need continuous updates, and to also consider a direction of receiving deal information through the event-based CDC platform Casita. Clearer logical separation between product information and deal information could further improve maintainability.