vLLM Introduces Push-Based KV Transfer to Cut TTFT by Up to 3x in Disaggregated Inference

vLLM has introduced a push-based KV transfer mechanism using the NIXL connector to optimize disaggregated inference. Unlike the traditional pull-based method, which serializes prefill completion, router signaling, and block allocation, the new approach parallelizes decode-side setup with prefill computation. This eliminates serialization bottlenecks and moves transfer setup tasks to a background thread on the prefill side, preserving latency-critical cycles during token generation.

Performance Improvements

Benchmarks conducted on Amazon EC2 P5en instances (specifically p5en.48xlarge with 8x NVIDIA H200 GPUs) using the Qwen3-32B model demonstrate significant gains under high load and long prompt conditions:

• TTFT (Time To First Token): Improved by up to 3x (e.g., reducing latency from ~1,744 ms to ~540 ms at 8 QPS and 16k input length).
• TPOT (Time Per Output Token): Improved by up to 30% (e.g., reducing latency from ~10.4 ms to ~7 ms at high concurrency).
• Load Dependency: At low load and short prompts, push and pull modes perform similarly. The benefits become prominent as request rates and input lengths increase.

Implementation and Trade-offs

The push protocol involves the decode side pre-registering target blocks with the prefill side. Once both prefill computation and registration are complete, the prefill side pushes KV blocks to the decode side via NIXL WRITE. This moves transfer setup tasks to a background thread on the prefill side, preventing them from blocking the decode instance's latency-critical token generation.

A key trade-off is that the push method requires fixing the decode instance selection earlier, which limits the flexibility of load-aware routing and QoS compared to the pull method. The implementation leverages EFA (Elastic Fabric Adapter) optimizations, utilizing GPUDirect RDMA to transfer KV blocks directly between GPU memory.

Verification Environment

The results were validated on a 1P1D (1 Prefill, 1 Decode) deployment with the following configuration:

• Hardware: 2x Amazon EC2 p5en.48xlarge instances (8x NVIDIA H200 each) connected via EFA.
• Software: Ubuntu 24.04, NVIDIA Driver 580.126.16, CUDA 13.0, vLLM 0.29.0, NIXL 1.4.1.
• Configuration: Prefix caching disabled to ensure full KV transfer measurement.

Future work includes applying the background NIXL threading model to the pull connector to improve its TTFT, as tracked in GitHub PR #45211.