vLLM Semantic Router Benchmark: 550ms Latency Reduction on Switching, 100% Success Rate with Pre-Boundary Selection

Conducted a benchmark of vLLM Semantic Router and Envoy-based agent model switching to analyze the performance and trade-offs of reactive switching policies
The reactive switching policy showed a 31.8%p reduction in the proportion of strong model usage and a 550ms reduction in average latency per session compared to the gate-disabled control group, but it failed on 3 final checks that the control group passed
In the native tool experiment, selecting the strong model before reaching difficult boundaries improved the final success rate from 37.5% to 100%, but at the cost of a 52.5%p increase in strong model turns and a 1,275ms increase in latency per session
Model A's KV cache cannot be reused by Model B, requiring a prefill; the study proposes a principle of not overriding model decisions based on cache hits for models other than the one designated by the semantic policy
Distinguished four types of conversation state—logs, tool ownership, provider management state, and KV cache—and proposed an architecture that separates the responsibilities of the Semantic Router (model selection) and the inference router (endpoint selection)
The cause of failure in enforce mode was analyzed as a sequence error occurring during final synthesis, where the strong model inherited an incorrect answer from the logs after the small model failed for 3 turns
New gates are recommended to first run in observe mode and prioritize setting quality requirements; tools with side effects require idempotency safeguards

vLLM Semantic Router Benchmark: 550ms Latency Reduction on Switching, 100% Success Rate with Pre-Boundary Selection

Key point

Details

Related