Ai2 Cuts Wait Times by 96% with New GPU Scheduling Policy

Allen Institute for AI (Ai2) announced that it revamped its GPU cluster scheduler, reducing p90 wait times for debug jobs from 2 hours to 30 seconds. To address GPU squatting and priority inflation issues in the previous priority-based system, they introduced 'GPU time budgets' and 'Hierarchical Fair-Share' policies instead of priorities.

New Scheduling Policy

Administrators allocate GPU time ratios according to research programs and project trees. For example, if Program A is allocated 70% and Project A1 50%, 35% of the total is assigned. The system tracks utilization within a 7-day sliding lookback window, prioritizing groups with unused allocations. Additionally, jobs must declare a minimum execution time (up to 8 hours) upon submission, during which they are protected from preemption.

Operational Results and Performance

Over 30 days of real-world operation, cluster utilization remained at 98% both before and after the change (with demand at 2–3 times supply), and 98% of the GPU time teams were promised was actually delivered (minimum 90% among 15 team allocations). Notably, p90 wait times for debug jobs improved from 2 hours to 30 seconds, and median wait times for the largest H100 cluster improved from 5 minutes to 24 seconds. Furthermore, automatic draining after minimum execution time ended reduced the number of human-intervened host repairs by 74%.

Remaining Challenges and Roadmap

For interactive development sessions, the application of the protection time cap (8 hours) increased inconvenience compared to the previous state (up to 1 week). Ai2 has included deploying a CPU-only cluster for data preparation and introducing resumable sessions in its roadmap to address this. Additionally, they are using a simulator to reproduce and measure the potential capacity fragmentation issue that may arise from reduced opportunities to interrupt multiple jobs simultaneously during large batch submissions.