Optimizing Amazon S3 Storage Costs and Image CDN Traffic

Inflab faced a problem of continuously increasing storage costs as it stored large volumes of data in S3, including original lecture videos, multiple resolution-encoded versions, audio normalization outputs, thumbnail sprites, and CDN access logs. Simple Lifecycle rules alone made it difficult to finely reflect object access patterns, and storage class transition costs and retrieval costs were also a burden.

As a solution, they introduced S3 Intelligent-Tiering. It allows class transitions immediately upon upload, and was especially well-suited for buckets where access patterns are hard to predict, there are many objects larger than 128KB, and a caching layer like CloudFront exists. By default, objects move to IA after 30 days of no access and to AIA after 90 days of no access, with a monitoring cost of about $0.0025 per 1,000 objects.

They also calculated the adoption cost in advance. Since transitioning existing objects via Lifecycle incurs a transition cost of $0.01 per 1,000 objects, pre-calculation became more important as the number of objects grew. New uploads were handled with put_object(..., StorageClass="INTELLIGENT_TIERING"), and for buckets with strong archival characteristics like original videos, they used CopyObject for batch conversion to reduce transition request costs.

The application method differed depending on bucket characteristics.

• New objects: Specify Intelligent-Tiering at PutObject
• Archive buckets: Batch conversion via CopyObject
• Encoding output buckets: Gradually apply Lifecycle rules based on file size

After 3-4 months of transition, costs for infrequently accessed objects decreased, and overall S3 costs were reduced by 38%. To identify object counts and sizes in advance, they also used S3 Inventory and Athena queries, first estimating the number of objects larger than 128KB and the expected transition cost before applying the changes.

For image CDN traffic optimization, they used a CloudFront + Lambda@Edge-based resizer. It converts images to the required size and format at request time, and runs only on cache misses, eliminating the need to pre-store original images in multiple sizes.

On the format side, they added AVIF to further reduce traffic. In thumbnail conversion tests, an original JPG of 100KB was reduced to 68KB in WebP and 55KB in AVIF. Since Sharp supports AVIF, applying it was as simple as adding avif to the list of allowed formats.

For browser compatibility, they configured fallbacks using the <picture> tag. If AVIF is supported, AVIF is served; if not, WebP; and if neither is supported, the original JPG is served. However, since AVIF encoding is slower than WebP and hardware decoding support is limited, fallback design on the frontend was important.

Animated GIFs had significant constraints for real-time conversion in Lambda@Edge. Sharp does not support animated AVIF, and Lambda@Edge has limits of 50MB package size, 1MB Origin Response size, and 30 seconds execution time, making it unsuitable for processing large GIFs.

So GIFs were pre-converted to MP4 (H.264) at upload time using a regular Lambda and FFmpeg. The flow is S3 upload → ObjectCreated event → Lambda execution → GIF download → MP4 conversion → S3 storage, and on the frontend they were played using the <video> tag. While this approach introduces an initial upload delay, it reduced CDN traffic in the long run and allowed lighter media to be served to users.