AI BriefingKurly Secures Search Indexing Pipeline Stability by Integrating Kafka Streams with Spring
Key point
Kurly integrated Kafka Streams with Spring to simplify message merging logic and transition to a structure enabling real-time health checks when threads die.
Details
Following a large-scale system overhaul, Kurly's Search/Recommendation Service team rebuilt a pipeline that combines and indexes messages from different topics into a single piece of data. Previously, the team used Redis as an intermediate store to load and merge data, but adopted the Kafka Streams library as a more effective and simpler alternative.
In the early stage of applying Kafka Streams, the pipeline was run via Spring's ApplicationStartedEvent, but an error in processing incorrectly formatted messages caused the Kafka Streams thread to terminate. Even in this case, the Spring Application itself remained in a normal state, making failure detection difficult.
To address this, the team changed the architecture by leveraging Spring Kafka's support for Kafka Streams. By applying the @EnableKafkaStreams annotation, they delegated the lifecycle management of the Kafka Streams client directly to Spring, allowing developers to focus solely on business logic.
They also implemented a HealthIndicator to build a real-time health check system. When the Kafka Streams state is CREATED, RUNNING, or RE-BALANCING, it returns Health.up(), and when it is ERROR, NOT_RUNNING, or PENDING_SHUTDOWN, it returns Health.down() along with detailed information. This ensures stability by allowing the application to be notified immediately when a thread dies.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.