Scaling vLLM Without OOM

In GRPO online RL for Jamba-family models, sharing an LLM-as-a-Judge(JLM) deployment across multiple training jobs can reduce GPU idle time, but it also exposes the system to unpredictable burst traffic. To address this, single-node performance optimization and multi-node scaling were applied together.

On a single node, Auto-Tune vLLM was used to tune configurations. Benchmarking was done with GuideLLM, and the parameter space was explored using NSGA-II from Optuna, following the sequence of workload analysis, optimization parameter setup, running 300 trials, and selecting a Pareto-optimal configuration.

Benchmarks were run under conditions of 2K input tokens (up to 8K), 100~1K output tokens, 2k concurrent requests, and a 5-minute timeout, with the following 5 objectives.

• Maximize Output tokens/second
• Maximize Requests/second
• Minimize Request latency
• Minimize Time-to-first-token
• Minimize TPOT (inter-token latency)

Of the 300 trials, 125 succeeded, and the rest ended in OOM or timeout, but the framework handled these gracefully. This left 24 Pareto-optimal configurations, and since total throughput was prioritized, the configuration with the highest token throughput was selected. The target model was a 32B dense attention model, and the hardware was H100 SXM 80GB.

The comparison results had to separate the effect of a simple version upgrade from the effect of configuration tuning. Simply upgrading vLLM v0.8.5 → v0.11.0 alone yielded about a 15% throughput improvement and 5% latency reduction under the same configuration, but the real improvement came from tuning. On the same 4 GPUs, the tuned configuration on the new vLLM produced roughly 2x throughput and 2x lower latency compared to the existing configuration.

The key was comparing at the deployment-unit level. Converting via num_instances = total_gpus / tensor_parallel_size and normalizing to 8 GPUs, tp=4 showed the best balance, and as a result, 2 instances of 4-GPU performed better than 8 instances of 1-GPU, 4 instances of 2-GPU, and 1 instance of 8-GPU. Presumably, tp=1 suffers from insufficient KV cache memory, while tp=8 suffers from reduced efficiency due to increased communication overhead.

For multi-node scaling, HPA criteria were chosen carefully. GPU utilization cannot distinguish between healthy usage and overload, and throughput metrics also fail to reveal backlog once saturation occurs. Instead, vllm:num_requests_waiting and vllm:request_queue_time_seconds, which represent vLLM's queue state, were examined, and the more intuitive average queue size was adopted as the autoscaling metric.

The HPA configuration was set to scale out when the average number of pending requests exceeded a threshold.

• minReplicas: 2
• maxReplicas: 10
• target queue size: 1000

The threshold of 1,000 pending requests per pod was set experimentally to match the point where queue wait time begins to affect the training job timeout. However, since it takes about 3 minutes for a new JLM instance to become ready, the threshold had to be set with the premise that scale-up can be fast but instance startup is slow. Scale-down was operated conservatively with a 15-minute stabilization window to prevent thrashing.