NVIDIA Dynamo Unveils Vision Encoder Disaggregated Serving Technology

NVIDIA has unveiled EPD (Encode-Prefill-Decode) disaggregated serving technology, which separates the vision encoder (ViT) from the prefill/decode processes and runs it as an independent worker during multimodal LLM serving. This is implemented via the NVIDIA Dynamo framework and has demonstrated a reduction in TTFT (Time to First Token) by up to 93% in benchmarks targeting the Qwen3.5-122B-A10B model.

Principles and Benefits of EPD Disaggregated Serving

Existing integrated serving causes bottlenecks as compute-intensive prefill, memory-intensive decode, and vision encoding compete on a single GPU. The EPD architecture separates the Encode Worker (E), which handles media decoding and ViT forward propagation, from the PD Worker, which handles LLM inference. This separates scheduling domains and efficiently transmits embeddings and KV blocks using NIXL.

Measured results show that under conditions of 10 image inputs and an output length of 1024 tokens, TTFT decreased by 50–58% compared to integrated serving, and goodput (throughput) improved by 1.70x under the same SLO. Notably, in mixed text and image traffic (50:50), head-of-line blocking was resolved, reducing text TTFT by 42.2% and image TTFT by 30.8%.

Optimization Conditions and Limitations

EPD technology is not beneficial for all workloads. The effect is most significant when ViT computation accounts for 5–11% of TTFT, with greater benefits seen when media load is high and output length (OSL) is short. Conversely, when media input is low and output is very long (OSL 512 or more), the collocated approach may actually worsen end-to-end latency (E2E Latency).

To address EPD's limitations, NVIDIA proposed three optimization levers:

• Parallel Media Decoding: Moving decoding to a Rust frontend reduced average latency by 26%.
• Embedding Cache: Utilizing a CPU LRU cache to reduce latency for repetitive media workloads.
• Multimodal KV Routing: Increasing KV block reuse rates through content hashing.

This technology is reproducible in vLLM 0.26.0 and SGLang 0.5.16 environments, and the related scripts and benchmark code have been released under the Apache 2.0 license.