AI BriefingOptimizing Gemma 4 Serving with vLLM on Amazon EKS: Cold Start Reduced from 428s to 226s
Key point
By leveraging Run:ai Streamer and cache optimizations for serving Gemma 4 with vLLM on Amazon EKS, cold start times were reduced from 428 seconds to 226 seconds.
Details
This case study introduces optimizations to address long cold start times when serving the Gemma 4 model in an Amazon EKS environment. In the default configuration, cold starts take 428 seconds, with primary bottlenecks being S3 weight loading and torch.compile cache invalidation.
Weight Loading and Compile Cache Optimization
Applying Run:ai Model Streamer resolved FUSE bottlenecks and reduced weight loading time from 127.6 seconds to 19.8 seconds (a 6.5x improvement) via parallel range reads. Additionally, using hostPath to persist the torch.compile cache and publishing the compile cache to S3 maintained compile times at approximately 6.6 seconds even on new nodes.
Image Pulling and Sleep Mode Utilization
Pre-pulling images via EC2NodeClass userData halved pull times. To handle intermittent demand, applying vLLM sleep mode moves GPU memory to CPU RAM, allowing a return to inference state in just 0.97 seconds. This is approximately 137 times faster than a warm restart.
Final Performance and Operational Strategy
Ultimately, cold starts improved from 428 seconds to 226 seconds (-47%), and warm restarts improved from 314 seconds to 133 seconds (-58%). Operational costs can be minimized by combining streaming and hostPath for frequent restarts, S3 caching and image pre-pulling for Spot instance replacements, and sleep/wake modes for intermittent demand.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Optimizing Gemma 4 Serving with vLLM on Amazon EKS [Part 2: Throughput and Latency SLOs on a Single GPU]
LLM Serving: The Difference Between Launching and Launching WellAmazon EKS Launches Direct Configuration for Managed Control Plane Parameters
vLLM Publishes Performance Benchmarks for 5 Speculative Decoding Methods on AMD GPUs