How to Deliciously Split and Process Hundreds of Millions of Records (with Partitioning)

While regenerating hundreds of millions of ledger statistics records, an OOM (Out of Memory) error occurred due to a structure that read everything in at once. The starting point for the solution was a two-stage partitioning strategy: first splitting the entire period into monthly units, then further splitting each batch internally into daily units for parallel processing.

Among Spring Batch's parallel processing methods, Partitioning was chosen because it clearly divides data within a single JVM. A Manager Step manages the overall partitioning, while Worker Steps each run independently with their own ItemReader, ItemProcessor, and ItemWriter. This structure reduced data contention while securing parallelism.

The core of Partitioning is the Partitioner and PartitionHandler.

• Partitioner: Divides the work scope based on criteria such as date ranges and creates an ExecutionContext.
• PartitionHandler: Executes partitions in parallel based on gridSize and TaskExecutor.
• gridSize, corePoolSize, and queueCapacity are tuned together to match parallel efficiency.

In practice, DateRangePartitioner was used to create daily partitions, connected to TaskExecutorPartitionHandler and ThreadPoolTaskExecutor for execution. queueCapacity was set to 0, and corePoolSize was matched to gridSize so that partitions would be processed in parallel immediately. When CPU usage was low, gridSize was increased; when memory or DB connection bottlenecks appeared, it was decreased.

However, even with partitions finely divided, problems arose again when each Worker loaded millions of records for a single day into memory all at once. So the reading stage was switched from MongoPagingItemReader to MongoCursorItemReader. skip()-based page queries become increasingly costly further in, but cursor-based streaming keeps memory usage low and maintains consistent performance even with large-scale data.

Ultimately, there were three key points.

• Split the target first using Partitioning.
• Reduce memory burden with a cursor-based ItemReader.
• Secure write efficiency with Bulk Write.

This shows that when dealing with large-scale data, simply running things in parallel is not enough — the partitioning method and the read/write strategy must be designed together to achieve stable processing.