vLLM-Omni Unveils Multimodal Unified Serving Architecture

The vLLM-Omni technical report (arXiv:2610.09307) has been released, describing a structure that serves text, audio, image, video, and robot actions within a single runtime. The code is available on GitHub under the Apache-2.0 license, adopting a hierarchical structure where an orchestrator handles request acceptance and stage progression, while execution engines process model components.

Core Architecture and Performance

The orchestrator handles request routing and session management, while execution engines handle token scheduling and denoising. Notably, the async_chunk mechanism passes chunks to the next stage immediately as they are generated by the previous stage, reducing TTFP (Time to First Audio Packet). Testing Qwen3-Omni in an H100 environment showed that disabling async_chunk at concurrency 32 increased TTFT by 6.7x and degraded RTF to 0.93, approaching the real-time limit (1.0).

In an H200 environment, applying the MRv2 profile reduces E2EL and RTF by approximately half compared to the default deployment at concurrency 16 or higher, and improves TTFP from 1,064.6ms to 500.4ms at concurrency 32. However, MRv2 has reported issues with some startup failures, requiring defensive coding.

Supported Models and Ecosystem

Currently supports 79 model rows, including AR Omni (Qwen3-Omni), full-duplex speech (PersonaPlex), TTS (CosyVoice3), image generation (FLUX), video generation (Wan2.1), world models (Cosmos3), and VLA (GR00T N1.7). It exposes an OpenAI-compatible API and the OpenPI protocol for robots, ensuring compatibility with existing serving environments.

Limitations and Caveats

Direct comparisons with external frameworks (such as SGLang-Omni) were excluded from this release. Some opt-in profiles exhibit uneven stability, and for text-only request paths, vLLM-Omni shows a TTFT approximately 53% higher than thinker-only vLLM, indicating overhead. Measurements were currently limited to NVIDIA GPU (H100/H200) environments, with support for other hardware such as ROCm/Ascend remaining a future task.