How to Achieve True Serverless GPUs

Modal believes that AI inference, being far more variable than training, is better suited to serverless. However, if spinning up a new replica takes tens of minutes, it's impossible to scale instantly with demand changes, and even loading a billion-parameter LLM on B200 with SGLang can be delayed for a long time just by GPU acquisition alone.

The core of the problem is GPU Allocation Utilization. This is defined as GPU-seconds running application code ÷ GPU-seconds paid for, and according to a 2024 survey cited in the article, many organizations stayed below 70% even at peak times, with actual values dropping as low as 10-20%. Fixed allocation struggles to withstand spikes, while over-provisioning only drives up costs.

There are four solutions.

• Cloud buffers: Keep a small amount of healthy idle GPUs on hand to immediately place new replicas.
• Custom filesystem: Lazily load container images from a content-addressed, multi-tier cache to reduce root filesystem build time.
• Checkpoint/restore: Skip CPU-side initialization by restoring memory state.
• CUDA checkpoint/restore: Skip GPU-side CUDA context initialization by directly restoring it.

Modal explains that this combination shortened inference server spin-up by about 40x, reducing it from around 2,000 seconds to around 50 seconds. As a result, even for inference workloads with volatile demand, supply can quickly keep pace, achieving a closer approximation to true serverless GPUs.