Building a Pinterest VLM Serving Stack Based on NVIDIA Dynamo

Pinterest built a large-scale Vision Language Model (VLM) serving stack based on NVIDIA Dynamo and Blackwell GPUs. This infrastructure is applied to various AI services, including Pinterest Assistant, hybrid search, and multimodal reranking, optimizing performance and cost by combining Qwen3-VL models with their proprietary PinCLIP embeddings.

Challenges and Architecture of VLM Serving

High Prefill costs due to image processing and KV Cache pressure from long conversation histories are the main bottlenecks in VLM workloads. To address this, Pinterest adopted an Encoder/Prefill/Decode (E/P/D) disaggregated serving architecture. By leveraging Dynamo's Rust-based router and LMCache, they implemented multi-tier KV Cache offloading between GPU, CPU DRAM, and NVMe, thereby managing VRAM pressure and reducing latency.

Performance Improvement via Projection Embeddings

By using pre-computed PinCLIP Projection Embeddings instead of traditional pixel-based inputs, they significantly reduced vision encoder computation costs. This approach achieved an 85x improvement in average TTFT (Time To First Token) and a 7.3x improvement in end-to-end latency. Additionally, when processing 250 embeddings, they can handle 25x more visual context with latency comparable to that of 10 pixel images.

Scalability and Operational Efficiency

Through Dynamo's LoRA Hot Loading feature, multiple LoRA weights can be dynamically loaded onto the base model without separate full deployments, reducing GPU usage and operational overhead. Using NVIDIA AIPerf for DAG benchmarking, they simulate multi-turn conversations and agentic workflows in actual production environments, while Dynamo Planner dynamically adjusts the number of prefill/decode replicas to meet SLAs.