vLLM Routing and KV (16 min read)

Mixed production traffic generated by FCK was used to compare vLLM V1, SGLang, llama.cpp, and TGI under the same OpenAI-compatible interface. The serving sweep tested a balanced pool, a large-token pool, a small interactive pool, prefix-cache routing, slow-client isolation, and class-aware routing.

There were 6 request classes.

• interactive chat
• RAG using repeated prefixes
• long-prefill requests
• agent tool loops
• batch summarization
• slow streaming clients

The best profile was vllm-v1/class-aware-router. It had the best balance of time-to-first-token (TTFT), inter-token latency (ITL), slow streaming client isolation, and useful throughput, and was far better suited to mixed traffic than a single global vLLM pool.

The operational criteria were simple.

• Bundle interactive traffic with a smaller max_num_batched_tokens and a moderate max_num_seqs.
• Separate long-context and batch jobs into a separate pool with a larger token budget.
• For chunked prefill, keep max_long_partial_prefills lower than max_num_partial_prefills.
• Treat max_num_batched_tokens, max_num_seqs, partial-prefill limits, and stream interval as workload control values.

In experiments at larger scale, a single shared pool failed to pass the TTFT/ITL gate, but class-aware routing accepted nearly all requests. The prefix-cache-only and slow-client-only profiles helped in their respective segments but were not a whole-system solution.

For builds from source, Debian required a larger rootfs and was blocked by GCC version gating. On Fedora 44, the combination of GCC 16, Python 3.14, VLLM_TARGET_DEVICE=cpu, MAX_JOBS=4, and numactl-devel successfully built the extension modules, and installing tcmalloc was recommended for runtime performance.

The Hybrid KV lab validated a PagedAttention rewrite path. While preserving block ownership, prefix sharing, refcounts, partial blocks, eviction, and copy-on-write, it exposed a compact logical span instead of a long per-token block table. The correctness oracle checked against odd lengths, MQA/GQA, ALiBI, sliding windows, prefix reuse, copy-on-write, partial final blocks, and FP8-style scaling, passing 30/30 on the first run, making virtual-contiguous and hybrid-prefix-shared the top candidates for hardware profiling.