GPU Job Scheduling Using Idle Inference GPU Pool

Since LLM services secure resources based on peak traffic times, a problem arises where massive amounts of GPU resources remain idle, occupying only memory, during periods of low traffic.

To solve this, we implemented sophisticated Auto-scaling utilizing vLLM's real-time throughput and Queue status metrics. This allows us to identify idle resources during nighttime hours while maintaining service stability.

The identified idle resources are allocated to research and experimental tasks in a Best-effort manner. If service traffic surges and GPU reclamation is needed, the system is designed to immediately halt running tasks, ensuring service availability as the top priority.

The GPU job pipeline has the following characteristics:

• Versatility: All tasks are defined as Docker image units, supporting various frameworks and environments.
• Scalability and Flexibility: The structure allows new task types to be added and run without separate development.
• Reproducibility: Execution settings are separated into parameters rather than code, ensuring reproducibility of experimental results in the same environment.

All tasks are organized into steps based on Argo Workflows, allowing the process from data preprocessing to training and evaluation to be managed by combining independent steps. All outputs are managed via Cloud storage, ensuring traceability of experiments.