Extending Ad Frequency Capping Real-Time Aggregation to 7 Days with Apache Flink and RocksDB Tuning

Frequency Capping is a core mechanism that accurately counts the number of ad impressions per user to prevent budget waste and lost impression opportunities. Short intervals were already handled with Flink, but extending real-time processing to long intervals required matching both aggregation accuracy and operational stability.

The existing structure was a batch-based design combining Airflow DAG and Redis lookups. At serving time, up to 4 Redis lookups were used to build the 7-day aggregation, and due to the batch nature truncated to hourly units, it was difficult to provide event-level precise sliding aggregation.

The solution direction was to treat State as the SSOT, creating a real-time structure that serves 1-minute to 7-day intervals through a single Redis lookup. To achieve this, the code was shared, but split into three Flink apps with different bottleneck patterns: minutes / hours / days.

• minutes: For the 1-minute~30-minute interval, Write Stall was the bottleneck due to frequent expiration handling, resolved by increasing managed from 500MB → 1200MB and WBR from 0.25 → 0.5.
• hours: As TTL grew longer and State became larger, Filter Block Cache Miss consumed CPU, improved with partitioned-index-filters=true, target-file-size-base=256MB, max-size-level-base=1GB, use-dynamic-size=true.
• days: For 7-day aggregation, checkpoint I/O was the bottleneck due to the larger scale of SST and Savepoint, with level optimization and large-scale memory allocation being key.

The most challenging problem was transition consistency. Backfill only raises counts without setting timers, while Catch-up re-reads the same events to restore both counts and expiration timers together. Putting these two into one pipeline would cause values to go wrong as timers fire first while historical data is still being accumulated, so the two stages had to be strictly separated.

At the transition boundary, three things had to align. The Redis write condition requires eventTime to be after the backfill end point, withIdleness was set to 60 seconds to prevent missing the MAX_WATERMARK right before the Bounded Source terminates, and the timerState TTL was kept sufficiently longer than the sliding window expiration so that decrement logic wouldn't be dropped even during failures or restarts.

At the operational stage, RocksDB bottlenecks were resolved separately per app. In minutes, WBM pressure caused Flush and Write Stall; in hours, Filter Blocks were repeatedly read from disk, consuming CPU; in days, the level structure itself had to be reduced. This is why, even with the same codebase, RocksDB settings had to be completely different.

Direct I/O was also applied commonly. In the Kubernetes environment, since OS Page Cache can conflict with the node's total memory, memory was operated centered on Block Cache controllable by Flink, via useOsPageCache=false and DirectIoRocksDBOptionsFactory.