Sharing Experience Building a Scalable Data Extraction Service

The existing data extraction method used a synchronous structure, so when large amounts of data were requested, users had to keep waiting in the browser, which led to server resource shortages and backend service overload.

To solve this, the team implemented an asynchronous response process and separated the backend server from the Excel generation server to distribute the load. They introduced Kafka to manage Excel generation events, and improved the system so that generated files are uploaded to S3, allowing them to be re-downloaded at any time.

To maximize data extraction efficiency, the team adopted the Partitioning approach of Spring Batch. Within a single Job, multiple Secondary partitions are created to split and process the data, producing several Excel files that are ultimately merged into one, thereby improving work efficiency.