Introducing Kafka Streams Windows

After reviewing whether to process the settlement process in batch or with Kafka-based event processing, we chose an event-driven architecture for near-real-time processing and scalability. However, due to the nature of order cutoffs and early-morning delivery guarantees, we needed to reliably group and process spike traffic that concentrated at specific times.

There were three core requirements.

• Publish source data collected in 5-minute intervals to Kafka
• Aggregate spike-type data by product code + a fixed time interval
• Immediately reflect non-spike items into the settlement table without aggregation

For this, we chose Kafka Streams' tumbling window. This was because events must not overlap, and the aggregation result must be published only once. As an example, we configured a flow with a 5-minute window and a 1-minute grace period, using groupByKey → windowedBy(TimeWindows.ofSizeAndGrace(...)) → reduce() → suppress() to emit only the final result.

The key concepts here are grace period and stream time. The grace period determines how late an event can arrive and still be accepted, while the actual window close and emission timing is determined not simply by event time, but by the stream time that Kafka Streams manages internally. In other words, a result isn't emitted just because the window's time has ended — the closing is only recognized once a subsequent event arrives and stream time advances sufficiently.

There was trial and error along the way. The first problem was that the data's creation time and Kafka's publish time differed, causing settlement dates to misalign at the midnight boundary. To resolve this, instead of using the Kafka record's publish time, we redefined event time using TimestampExtractor to use the creation time.

The second problem was cases where no new events arrived at all after the window's end time. Sending dummy events via forward() inside the Processor did not change the stream time, and forcibly emitting by directly scanning the WindowStore caused duplicate emissions because the internal state was not recognized as closed.

Ultimately, we solved this by injecting dummy events into the topic from outside to forcibly advance the stream time. Since Kafka Streams' stream time is independent per partition, we had to either vary the key per partition or specify the partition number so that all dummy events wouldn't concentrate on a single partition. In the Processor, we ran a punctuation every 3 minutes based on WALL_CLOCK_TIME to emit dummy events, which allowed each partition's window to close properly.

In conclusion, this work was not simply a matter of slicing time, but a design that precisely defines the relationship between time and events. Window-based settlement must also account for boundary-time handling and recalculation policy, and this reaffirmed that in Kafka Streams, time only flows through input events.