The Current State of Model Serving Routing

Netflix treated Model Serving not as a simple score call but as an end-to-end workflow including preprocessing, feature computation, and postprocessing. As of 2025, the platform operates hundreds of model types and versions, handling 1 million requests per second.

Clients only need to pass common request context such as userId, country, device, and domain context. Models fetch the raw facts they need from other microservices through the platform to build features, and offline training uses ML fact store snapshots.

To hide this complexity, Netflix placed Switchboard as the central entry point and essential interface. A generic API Gateway or service mesh alone could not fully handle gRPC, experimentation integration, domain-specific context, and model lifecycle controls like shadow/canary/rollback.

The core consists of two abstractions.

• Objective: An enum representing a specific business use case that decouples clients from concrete models.
• Switchboard Rules: JavaScript configurations that let researchers define default models, A/B experiments, traffic splits, and gradual rollouts.

As a result, Switchboard routes traffic to the appropriate model and cluster shard by reflecting user state, locale, surface, A/B allocation, and client overrides. Clients don't need to worry about model version changes or VIP address changes, and researchers can rapidly iterate on experiments and deployments through a single central platform.