GPU Job Scheduling Using an Idle Inference GPU Pool

They built a pipeline that automatically allocates idle GPU resources arising during LLM service operation to research and experimentation, improving infrastructure efficiency.

Core Solution Approach

• vLLM Metric-Based Auto Scaling: Instead of CPU or memory utilization, resources are precisely adjusted based on the real-time throughput and queue wait status provided by vLLM, the LLM inference engine.
• Best-Effort Scheduling: Research jobs run during non-congested periods such as nighttime, but if service traffic increases, jobs are immediately halted and resources returned to the service to maintain stability.
• Argo Workflows Pipeline: Jobs are defined as Docker images, automating the entire process step by step from data preprocessing to training and evaluation.

Operational Results

• Resource Acquisition: Secured approximately 95,000 GPU hours over 3 months, which, converted to 24-hour terms, is equivalent to newly deploying about 55 GPUs.
• Cost Savings: Compared to using public cloud, this achieved a cumulative cost savings of approximately KRW 185 million over 3 months.