Redis Stream Adoption Case Study

The Data Product team has been ingesting and processing view, event, and impression data collected by the montelena receiver and post Processor, using it for metrics and analytics.

To identify unique active users, the team stored session_id history, but at moments of surging inflow such as Big Smile Day(BDS) or UTC Push, data processing delays occurred. There was a need to separate the load-generating logic into a separate application, and Redis Stream, which could reuse the existing Redis, was chosen as the alternative.

The reasoning was clear. Newly introducing Kafka or an MQ would increase development effort and operational resources, and the development timeline was short due to the event schedule. Since Redis was already being used as the session_id store, a structure was needed that could quickly ingest history without significantly touching the existing logic, and Redis Stream fit the bill, since it enables duplicate-free processing across multiple pod environments via consumer group.

Compared with Redis Pub/Sub, the difference is significant.

• With Pub/Sub, if there is no subscriber or the application encounters an issue, messages are lost.
• If there are multiple subscribers, the same message is delivered to all of them, risking duplicate processing.
• Redis Stream manages messages based on the last received record id, using commands such as XADD, XREADGROUP, XACK, XPENDING, and XCLAIM.
• It supports consumer groups, allowing a single stream to be divided and processed in parallel by multiple consumers, with ack and pending reprocessing also possible.

The implementation was a simple structure. The Publisher wraps the session object processed in the post Processor into an ObjectRecord and publishes it to streamKey, using the recordId returned by opsForStream().add() as the unique message id. The value type also supports MapRecord, but in this case a JSON string-based ObjectRecord was used.

The Consumer added spring-boot-starter-data-redis and implemented StreamListener. It created the consumer group in afterPropertiesSet(), configured StreamMessageListenerContainer, then started subscribing with ReadOffset.lastConsumed(), and called acknowledge() to signal completion after processing a message. A method of deleting based on recordId when an exception occurs was also used together.

There was also a point considered most important for operations. Redis Stream has no partition concept, so a single stream is processed in parallel by multiple consumers, and to evenly distribute messages across a Redis cluster, additional design is needed to split multiple streams by node. This also meant accounting for the fact that produce order is not strictly guaranteed.

Also, since it is an in-memory store, memory management is key. If pending messages that haven't received XACK are left unattended, Redis cluster memory keeps growing, so logic was added to run a scheduler every 1 minute to check the pending summary and reprocess, preventing memory full issues.

As a result of the application, after adopting Redis Stream, the traffic processing delays that occurred during UTC Push sends were resolved, and the service could be operated stably even during the BDS event period. This shows that Redis Stream is a valid option when real-time data processing needs to be quickly separated in a distributed environment.