Improving Large-Scale Data Processing for a MyData Platform

Kakaopay's MyData platform faced data processing limits in its existing system due to the rapid growth of its asset management service. In particular, the scale of the 'user's financial data collection history,' which must be retained for 3 years under the Credit Information Act, grew faster than expected, making architecture redesign necessary.

The existing system had three major problems. First, high QPS (Queries per second) caused DB replication delays, and even attempts to buffer the load via Kafka failed to keep up with the incoming data volume, worsening Kafka Lag. Second, storage capacity shortages were anticipated due to accumulating data. Third, the surge in log data lengthened statistical batch execution time, adding further load on the DB.

To address these issues, Kakaopay introduced the following improvements.

• Distributed data storage: Separated service data from historical data, and applied application-level Modular Sharding to the historical data to distribute both load and capacity.
• Data archiving: Shortened DB retention periods and moved older data into files for storage, resolving the DB capacity issue.
• Adoption of Palsonic: Moved away from computing statistics directly on the RDB, instead using Palsonic, an internal Hadoop-based system, to perform efficient statistical calculations.