AI Briefing
KOSign in

vLLM Router: Inference Request Routing Considering Sessions and Caches

·2026.11.12 20:00

Key point

vLLM Router provides routing functionality that distributes inference requests while considering sessions and caches.

1 / 3

Details

  • The Rust-based vLLM Router, forked from SGLang Model Gateway, distributes inference requests considering sessions, worker load, and prompt caches.
  • The default policy, cache_aware, selects workers based on input prefix match and load, and supports various policies such as consistent_hash and power_of_two.
  • It supports prefill/decode disaggregated routing, allowing different policies to be applied to each stage, and provides configuration examples for NIXL, NCCL, and Mooncake connectors.
  • It features operational capabilities such as circuit breakers, retries, automatic Kubernetes worker discovery, and Prometheus metrics collection.
  • It can be installed via pip install vllm-router on Python 3.8 or later and uses the Apache-2.0 license.
vllmkuberneteskv-cacherustload-balancinginference-router
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

PyTorchKRModular LLM Inference Handbook, 57 pages released for freePyTorchKRvLLM Semantic Router Benchmark: 550ms Latency Reduction on Switching, 100% Success Rate with Pre-Boundary SelectionRedditTandem-vLLM 0.2.0 Enables DeepSeek-V4.1-Flash on Ampere, Ada, and Hopper GPUsRedditIsoKron Research: MCP Servers and APIs Lose Concurrent Writes in Claude Opus TestsHacker NewsNVIDIA Tutorial: Serving GLM-5.3 NVFP4 on Dual DGX Stations with vLLMGeekNewsArcee releases nac, an open-source agent harness for long-running tasks