Kurly Resolves Search Indexing Rebalancing and Improves Throughput by Adopting Kafka Batch Listener

Kurly's search service, while generating indexes by combining product information and supplementary information, experienced rebalancing and message backlog issues caused by processing delays in the Kafka consumer. Initially, Redis was used to combine data, but the consumer failed to process 500 records within 5 minutes, causing group reformation, which made normal search service impossible.

To solve this, max-poll-records was first set to 1 to temporarily control rebalancing, but for a fundamental performance improvement, the team switched to a Batch Listener. Through the spring.kafka.listener.type: batch setting, the consumer was changed to process the polled list of records all at once, which was implemented via BatchMessagingMessageListenerAdapter.

This change made it possible to utilize the search engine's Batch indexing request feature. By switching from a method that requested records one by one to a method that gathers multiple records and requests them at once, the speed of message consumption and index processing increased dramatically. As a result, the same amount of messages could be processed in less time, securing stability such that rebalancing no longer occurred even with the existing settings.