Building a Flink-Based Stream Join Service for AI Real-Time Recommendation Systems

Azar's real-time recommendation system needed to combine user events generated at different points in time to reflect the latest information. As some events were lost and the combination rules became more complex, a stream join service was needed instead of a simple pipeline.

The platform compared Spark Streaming, Kafka Streams, and Apache Flink, and decided on Flink. Low latency, precise time control, state management, and support for Exactly Once processing were the key reasons, and among these, they chose KeyedProcessFunction, which enables complex time control.

In KeyedProcessFunction, state was managed per key, and the TimerService was used to register, extend, and cancel timers to determine the combination result. When all events arrived, the result was published immediately; when only some events arrived, partial publishing occurred upon timer expiration, and if a required event was missing, it was treated as a miss. On top of this, heartbeat events were added to wait a bit longer for necessary events, increasing the combination success rate.

On the deployment side, Savepoint was used to safely restore state, Kafka offsets, and timers, and the problem of timers expiring right after recovery was addressed using CheckpointedFunction and the recovery-time timestamp. Zero-downtime deployment was automated and solved using a Blue-Green strategy and Spinnaker Pipeline.

For the duplicate delivery problem, they initially considered Kafka Sink's Exactly Once and a separate dedup Flink application, but ultimately settled on Redis-based deduplication. Using SET NX or a Lua script, they prevented duplicates without concurrency issues even across two independent services, and deduplication latency dropped significantly from an average of 300ms to under 3ms.