[if(kakaoAI)2024] Kakaopay Securities' Intelligent Kubernetes Resource Optimization (Dr.Pym Project Sharing)

Kakaopay Securities' DevOps team pursued the intelligent resource optimization project Dr.Pym to address three key challenges arising from Kubernetes operations: resource shortages, degraded service stability, and difficulty in rightsizing.

Dr.Pym aims to flexibly adjust services' resource sizes according to the situation, achieving both cost savings and service stability at the same time. The project proceeds in a total of 3 phases, and it is currently focused on Phase 1, which provides resource recommendation functionality.

The core of Phase 1 is calculating recommended CPU and memory usage using Prometheus metrics.

• By leveraging Prometheus Recording Rules, the team prevented load caused by large-scale queries and increased data collection speed.
• The CPU recommendation selects the top 99% value via the quantile_over_time function, allowing for temporary spikes while preventing excessive resource allocation.
• The Memory recommendation is calculated by adding a margin of 15% for development environments and 20% for production environments to the maximum value over the most recent 2 weeks.