[if kakao 2022] The Best Reader Considering Batch Performance

Over 80% of batch processing performance is determined at the ItemReader stage, where data is read. In particular, the task of selecting only the necessary data out of billions of records is the key to overall performance.

When processing large volumes of data, Chunk Processing, which divides data into fixed units for processing, is essential to prevent memory limits and system overload.

The existing PageItemReader uses MySQL's LIMIT OFFSET approach, which has a performance limitation where query speed drops sharply as the Offset value grows larger. To solve this, using the ZeroOffsetItemReader, which retrieves the next data based on the last ID value, keeps the Offset at 0 and secures fast performance.

There is also the Cursor approach, which fetches data little by little. However, since JpaCursorItemReader loads all data into memory and can cause OOM (Out of Memory), it is recommended to use JdbcCursorItemReader or HibernateCursorItemReader instead.