Kurly Resolves DB Connection Leak and OOM Issue That Occurred After Changing MySQL Connector

After the Kurly team changed the JDBC Driver from MariaDB to MySQL, periodic Out Of Memory (OOM) occurrences and abnormal terminations occurred on AWS EC2 instances. It was initially mistaken for a traffic surge, but Heap Dump analysis confirmed that the AbandonedConnectionCleanupThread was excessively occupying memory.

Cause and Analysis

The core of the memory leak was in the MySQL connector's AbandonedConnectionCleanupThread. This thread is responsible for cleaning up network resources of connections that were not explicitly closed. To connect quickly to the Slave during DB Failover, the Kurly team set max-lifetime to 50 seconds, much shorter than the default (30 minutes).

However, this thread operates as a single thread and involves network I/O such as TCP/IP socket connection checks. Because the short max-lifetime caused the Connection creation rate to exceed the CleanupThread's processing rate, unprocessed PhantomReference objects kept accumulating in connectionFinalizerPhantomRefs, causing a memory leak.

Solution

The Kurly team upgraded to mysql-connector-j 8.0.22 or higher and disabled the thread by adding -Dcom.mysql.cj.disableAbandonedConnectionCleanup=true to the JVM options. In typical environments using HikariCP, the connector pool manages network resources, so disabling this thread is safe, and the system has been operating stably since the deployment.