Pinterest Introduces Event-Time Partition Finalization for Stream-Based DB Ingestion

Pinterest’s next-generation database ingestion framework, which migrated from batch to stream processing using Kafka, Flink, and Iceberg, introduced a partition finalization mechanism to solve data completeness issues. In the previous batch system, finalization was implicit upon job completion. However, in the new stream-based architecture, Flink writes change data capture (CDC) events continuously, meaning partitions can receive late-arriving data after their wall-clock hour has passed. This creates a challenge for downstream consumers who need to know when a partition is "finalized" and safe to read.

Event-Time Based Watermarking

The solution relies on an event-time based approach rather than wall-clock time. The Flink-to-Iceberg sink extracts the event time from each record and collects statistics, including minimum, maximum, and percentiles (e.g., 99th and 95th), between checkpoints. These statistics are stored in the Iceberg snapshot summary using a t-digest sketch, which approximates the distribution in a fixed, small footprint (a few hundred bytes) regardless of record volume. This sketch is mergeable, allowing parallel subtasks to combine their statistics efficiently at commit time.

Implementation via Extensible Sink

The logic is implemented through two generic extension points in the Flink-to-Iceberg sink: Accumulators and CommitProcessor.

• Accumulators: Lightweight aggregators running inside writer subtasks that track event-time statistics per checkpoint window. Their snapshots are merged by the committer and written to the snapshot summary.
• CommitProcessor: Hooks that run before and after each Iceberg commit. The post-commit hook reads the merged statistics, calculates a partition finalization watermark, and updates the Iceberg table property.

The watermark algorithm typically takes the minimum of per-commit minimum event times across recent commits and rounds down to the hour boundary. It is monotonically increasing, ensuring that a finalized hour remains finalized even if late data arrives, which triggers an alert to the consumer.

Propagation and Generalization

Since the finalization signal is stored as an Iceberg table property, it propagates easily. The Spark upsert job reads the watermark from the CDC table and writes it to the base table during its commit, allowing downstream jobs to gate their processing via standard metadata sensors without needing external services. The mechanism is configurable, letting tables choose between conservative completeness (using minimums) and freshness (using percentiles). This architecture is not limited to DB ingestion; it is being adopted for Pinterest’s next-generation Stream Ingestion framework.