Reducing GPU Cold Start Time via gVisor Snapshotting

The Cold Start problem that occurs when running AI models in production causes delays of several minutes due to model weight loading, CUDA initialization, kernel compilation, and other factors, leading to resource over-provisioning and increased costs.

To solve this, Cerebrium implemented CPU and GPU memory snapshotting (Checkpointing) technology within a custom gVisor-based runtime. This technology works by saving a snapshot of a fully initialized container state (including model weights, process state, and compiled kernels), then instantly restoring it when a new container is requested.

Key features and results are as follows:

• Ultra-fast restoration: By skipping the existing image download and initialization process and directly restoring from the snapshot, cold start times were reduced by over 80% for certain workloads.
• vLLM compatibility: Optimized so that real CUDA workloads such as vLLM can be restored stably and quickly.
• Infrastructure efficiency: By dramatically reducing the time required for model loading, GPU resources can be scaled more flexibly and costs can be optimized.