Making User Sequence Data More Cost-Efficient, Faster, and Easier to Use

User sequences are ordered lists containing recent relevant events and rich signals (enrichments), and they are core data for ranking, search, and recommendation systems. However, the existing approach had issues: the process for generating training data and real-time serving data differed, causing data inconsistencies, and adding new signals required building complex pipelines each time, which was costly.

To solve this, Pinterest introduced a core principle called 'One Definition, Many Runtimes'. Once a signal or event type is defined just once, it can be used identically across three different workloads: real-time indexing, batch indexing and backfill, and online serving.

The platform consists of the following 6 core components:

• Ingestion: Stream and batch data ingestion
• Enrichment and execution layer: Data transformation through a shared execution engine
• Real-time indexer: Streaming jobs for low-latency updates
• Batch indexer and backfill pipeline: Historical data processing
• Columnar, time-partitioned storage: Efficient reads and storage
• Online serving API: Providing sequences based on client requests

In particular, by adopting a Configuration-as-Code approach, instead of building an entirely new pipeline when adding a new event or signal, deployment can be done quickly through Python-based configuration changes alone. This maintains data consistency while reducing operational costs and dramatically increasing development speed.