Netflix's In-House LLM Serving Infrastructure

Instead of using external LLM APIs as before, Netflix built a complete LLM serving system on top of its own ML platform. A JVM-based unified serving system handles routing, A/B testing, candidate generation, feature extraction, and inference all in one place. Small models run in-process, while large models that require GPUs are delegated to the Model Scoring Service (MSS).

Engine Selection: Adopting vLLM

Initially, Netflix used TensorRT-LLM, but by mid-2025 the performance gap between open-source engines had narrowed, and workloads had also expanded. After rebenchmarking, Netflix chose vLLM as its primary engine. The reasons: it can load custom model architectures without a compilation pipeline, it provides extensibility hooks for custom decoding logic, it's easier to debug, and it's already widely used in the research community, keeping research-to-production costs low.

Triton Integration and Packaging Strategy

Triton supports two packaging approaches. The Python backend requires explicitly defining I/O tensor specs, while the vLLM backend dynamically generates them at deployment time using only JSON configuration. The vLLM backend is ideal, but in practice Triton/vLLM version mismatches and custom model logic issues occurred. For models that aren't standard HuggingFace-compatible, the Python backend must be used.

API Design: OpenAI-Compatible Interface

All models, from XGBoost to large-scale LLMs, are served through the same gRPC calls, with an OpenAI-compatible HTTP API provided as an additional frontend. This minimizes code changes when switching from hosted models to fine-tuned self-hosted models. Netflix adopted NVIDIA Triton's OpenAI-compatible frontend, but since there was a bug where response_format was silently dropped, they patched it via git-subtree and connected it to guided decoding.

Deployment Strategy: Red-Black vs Versioned

GPU deployments start more slowly than CPU services, and I/O schema changes create coordination problems. Red-Black deployment runs the new version and existing version in parallel, gradually shifting traffic, but coordination gaps arise when schemas change. Versioned deployment maintains independent instances for each model version, decoupling model deployment from consumer updates, but GPU costs increase during the transition period. Netflix recommends embedding variable configuration into models to avoid schema changes and using Red-Black.

Operational Considerations

When starting vLLM-on-Triton, downloading directly from S3/Hugging Face causes significant cold-start latency, so models are pre-cached on Amazon FSx. On the Prometheus metrics side, vLLM exports to a .db file while Triton exports via its own endpoint, but Triton's bridge only surfaces 9 of the more than 40 vLLM metrics. Netflix merged both metrics into a single /metrics endpoint using an HTTP proxy.

Constrained Decoding

Some of Netflix's production workloads require fine-grained control over token generation. Rather than applying business logic after inference and retrying, they use vLLM's custom logic extensions to embed constraints directly inside the decode loop, producing compliant output.