500MB Memory Spike Every 2 Hours After EKS Migration: Cause Identified as Log Compression and Page Cache

After migrating the API from EC2 to EKS, a phenomenon occurred where memory surged by 500MB every two hours. This issue progressed silently without error logs or OOM (Out Of Memory) records, making it difficult to identify the cause.

Initial Suspicions and Failures

The first suspect was native memory outside the Java heap. By modifying the external communication buffer management method and System.gc() options, we succeeded in reducing physical memory usage from 1,543MB to 959MB. However, the memory spikes did not disappear.

Java Version and GC Changes

Secondly, we upgraded from Java 21 to 25 and switched to Generational ZGC. Although it passed load testing in the staging environment, the problem recurred after production deployment.

Final Cause and Resolution

Ultimately, the cause was log gzip compression and page cache. In the EKS environment, where logs are stored in pod temporary volumes and collected by sidecars, the cache memory generated during the compression process affected the container limits. To resolve this issue, we adjusted the log processing method and readjusted the memory limits.