HyperCLOVA X 8B Omni Serving Deep Dive: From Architecture Design to Performance Optimization

HyperCLOVA X SEED 8B Omni is an omni-modal model that handles not only text but also images and audio, so in a service environment, role separation mattered more than a single unified architecture.

The serving architecture separated Encoder, Decoder, and LLM into independent servers. Bundling them into one server would occupy GPU resources even for unused features and cause components to compete for resources with each other, degrading performance.

For orchestration, a central Orchestrator approach was chosen. Compared to API Gateway-based Service Chaining, there is a risk of SPOF, but since the flow is concentrated in one place, debugging and operations become easier, securing development speed, while the Orchestrator's responsibilities were kept to a minimum.

The interfaces between components were divided by role.

• The LLM server and Orchestrator used the industry-standard OpenAI Chat Completion API as is.
• The Encoder converted images and audio into token IDs or embedding vectors.
• The Decoder received the token IDs generated by the LLM and produced the final images and audio.
• Encoder embeddings were passed to the LLM server via the Data Plane instead of an internal model plugin, simplifying the architecture.

Transferring intermediate outputs was solved using Object Storage (OBS) as the Data Plane. Encoder results were serialized into files and uploaded, and instead of passing the actual data, only address information such as an Object Key or Presigned URL was passed, allowing multiple servers to reuse the same data.

llm-d was introduced for distributed inference optimization. The core idea was to route requests to the server holding the KV cache to increase reuse rate, but this was not easy to apply since entry points were previously distributed across model servers and there was no body-based routing.

To achieve this, the architecture was changed as follows.

• Model server entry points were unified into a single Gateway to apply common routing.
• EnvoyFilter and HttpRoute were used to convert model information in the request body into headers, and InferencePool was selected based on this.
• Server selection combined kv-cache-utilization-scorer(3.0), queue-scorer(2.0), and max-score-picker(2.0) to consider both cache reuse and load balancing together.

As a result, throughput improved by 2.1x, and KV cache utilization rose from 25-45% to over 90%. Improvements made during this process were also contributed directly to open source.

Component-level compute optimization followed as well.

• For the Vision Encoder, FlashAttention2 was enabled after fixing the dtype, and then the bottlenecked PatchEmbed was improved, changing 784 calls to Conv3D into a single Conv3D call.
• As a result, the Vision Encoder's forward pass improved by about 4x.
• The Vision Decoder, being Diffusion-based, had long sequences and repeated computation as bottlenecks, and it was confirmed that reducing the step count from 50 to 25 showed no significant quality difference, cutting the amount of computation in half.
• Next, USP (Ulysses Sequence Parallelism), suited for long fixed sequences, was applied, configuring parallelization with just two All-to-All communications before and after attention.

Overall, this serving design is summarized as a case that separated the complex input/output paths of an omni-modal model into a simple structure, and combined Object Storage-based data transfer, llm-d-based KV cache reuse, and Encoder/Decoder compute optimization to simultaneously boost throughput and efficiency in actual service.