Spotify and NVIDIA Collaborate to Optimize LLM Inference for Recommendations

Spotify is transitioning from traditional multi-step recommendation systems to Large Personalized Language Models to deliver real-time, personalized content across its platform. This shift, initiated with LLM-powered podcast recommendations in January 2026, requires efficient inference architectures to handle complex tasks like beam search and candidate scoring at scale.

Inference Challenges and Solutions

Initial implementations relied on vLLM for beam search and SGLang for candidate scoring. However, vLLM’s beam search implementation operated outside the core GPU engine, causing CPU bottlenecks and preventing full utilization of high-end GPUs. To address these inefficiencies, Spotify partnered with NVIDIA to unify these features within TensorRT LLM (TRT-LLM).

Key Technical Contributions

The collaboration resulted in several critical updates to TensorRT LLM, including:

• Beam Search Support: Integration of beam search into the sampler to reduce host overhead and improve GPU utilization, released in v1.2.0rc3.
• Out-of-Tree Models: Support for custom model plugins, enabling Spotify to implement a Qwen3 extension for soft token injection, released in v1.2.0rc4.
• Multi-Item Scoring: A Python API extension for efficient scoring of multiple candidates in a single request, equivalent to SGLang’s functionality, released in v1.3.0rc20.
• Multi-Modal Embeddings: Support for passing embeddings via image_embeds, released in v1.3.0rc0.

Performance and Personalization

Spotify employs soft tokens for personalization, projecting existing user embeddings into the LLM’s embedding space. This approach, based on the "Embedding to Prefix" method, allows for personalized recommendations without the latency penalties of large text prompts.

Benchmarks using the Qwen3 1.7B model demonstrate significant efficiency gains:

• Beam Search: Up to 54% cost reduction.
• Scoring Workloads: Up to 7% cost reduction.

These improvements enable Spotify to expand generative recommendation capabilities across more surfaces while maintaining control over latency and infrastructure costs. Future work will focus on quantization and optimizations for NVIDIA Blackwell architectures.