Reducing Image Pull Time with Image Caching on EKS Bottlerocket

Inflab used GB-scale large container images in its ML/data pipelines, and especially in environments with frequent node recreation like Spot instances, image pull and decompression became a bottleneck. Container preparation alone took up to 15 minutes, and it also unnecessarily increased EC2 costs and network costs.

As a solution, they applied a method of caching the data volume (EBS) where images are stored as a snapshot in an EKS + Bottlerocket environment. Since Bottlerocket separates the OS volume from the container data volume, restoring a snapshot of the volume containing the images to another instance allows the same cached state to be reused immediately.

Snapshot creation was handled using the script from bottlerocket-images-cache, AWS's official example repository. The flow is as follows.

• Create Bottlerocket EC2, Role, InstanceProfile, LaunchTemplate, etc. with CloudFormation
• Enable the admin container, then pull the necessary images with ctr
• Remove existing images and unnecessary services, then create an EBS snapshot of the /dev/xvdb volume that contains the cache

In this process, access to Bottlerocket was done through the control container and the admin container. With SSM permissions, you can access via the control container, and tasks like image pulling were performed in the admin container, which has root privileges and host filesystem access.

At the node provisioning stage, the created snapshotID was attached to blockDeviceMappings in the Karpenter EC2NodeClass, so that new nodes could use the cached images right from launch. As a result, they reduced the first container start time to an average of under 10 seconds.

However, snapshot-based volumes could experience read latency for large files on first access due to lazy loading. To reduce this, they examined three options: pre-loading with fio, raising IOPS/Throughput, and fast snapshot restore, and chose raising IOPS/Throughput after weighing cost versus effect. Fast snapshot restore costs around $540 per az per month, which was a heavy burden for their current usage pattern.

In summary, given short task durations and the cost structure of high-spec instances, it was more economical to initially raise EBS performance to load snapshot data quickly. For pipelines that frequently need to re-upload large images, or Spot environments with frequent node recreation, an image-caching snapshot strategy can be a valid optimization.