The Key to Speed and Cost: 'Image Token Imbalance' Eroding Multimodal AI Training

Efficiently utilizing GPU infrastructure in Sovereign AI development is directly linked to massive cost savings. According to Meta's data, pre-training Llama 4 required a cumulative 7.38 million H100 GPU hours, and even a 1% improvement in training speed can lead to cost savings worth hundreds of millions of won.

While scaling multimodal models on its own cluster, Team Naver discovered inter-GPU computational load imbalance, where training speed did not increase as expected despite the growing number of GPUs. When some GPUs were assigned data with many images and became delayed, a Straggler problem occurred where thousands of GPUs waited at the synchronization point, All-reduce.

A VLM consists of an LLM backbone that processes text and a Vision Encoder that converts images into image embeddings. While the LLM's computational load depends on the total sum of image tokens and text tokens, the Vision Encoder's computational load is proportional only to the number of image tokens, so the variance in image tokens per GPU significantly impacts overall processing speed.

The following methods are used together for large-scale training:

• Data Parallelism: Multiple GPUs split and process data, then sum the Gradients via All-reduce.
• Gradient Accumulation: Gradients from multiple Micro-batches are collected to update weights all at once.
• Sequence Packing: Samples of different lengths are packed tightly into fixed-length sequences to reduce computational waste caused by padding.

Since the result remains the same regardless of the order of Gradient summation, the batching of Micro-batches included in a single weight update can be reconfigured. Based on this, Team Naver efficiently packed image tokens and redistributed the load across GPUs, improving the overall training throughput of a 30B MoE multimodal model by 50.2%, and achieving a 13.3% improvement solely through packing and redistribution that resolved image token imbalance.