Vercel Sandbox snapshot optimization

Vercel Sandbox initially focused on reliability, but once it stabilized, performance became the bottleneck. The p75 for snapshot restore exceeded 40 seconds, and the team brought it down to under 1 second by improving download, decompression, and caching in turn.

Vercel Sandbox runs on the same infrastructure as Hive, an internal build product, and each sandbox is an isolated container inside a Firecracker microVM. A snapshot is a compressed copy of the sandbox disk, using the original .img format and .vhs (Vercel Hive Snapshot), a compressed format for upload/download. sandbox.snapshot() compresses .img into .vhs and uploads it to S3, while Sandbox.create() downloads it and decompresses it back into .img.

The original restore path was fully sequential.

• Download the entire .vhs from S3 in one shot
• After the download completes, decompress on a single thread

This structure caused delays ranging from a few seconds to tens of seconds for snapshots ranging from hundreds of MB to several GB.

The first improvement was parallel downloading using the Range HTTP header. Using transfermanager from the AWS Go SDK, chunks were split and downloaded concurrently, and after tuning, download speed improved by 2-5x. Next, decompression was parallelized by distributing each frame of the .vhs across multiple goroutines, and this step alone made restore 2-4x faster.

But the downloaded data was still written to disk first before decompression began. By streaming the S3 range request directly into decompression and eliminating the intermediate file, end-to-end restore time was cut by another 2x.

That wasn't the end of it. Since there was originally no fast path at all and every request was a cache miss, the team introduced a local disk cache using the large NVMe storage on metal instances. The cache stores the decompressed .img directly rather than the compressed version, and applies LRU eviction based on total disk usage rather than entry count. Since many customers reuse base snapshots repeatedly, this achieved a 95% cache hit rate, and on a hit, only the time to spin up the microVM and container remains.

As a result, p75 dropped from 40 seconds to sub-second, and p95 dropped from 50 seconds to 5 seconds. Going forward, there's also the option of cache affinity to route popular snapshots to machines that already have them cached, but since this could concentrate load on specific machines, it's being carefully considered. This optimization also applies to Automatic Persistence, currently in beta, which snapshots the filesystem when a sandbox is stopped and restores it instantly when it's started again.