Kakao Details VM Density Testing for Efficient VMware Infrastructure

Kakao conducted a comprehensive test to determine the optimal VM density for VMware infrastructure, focusing specifically on CPU performance. The goal was to identify the maximum number of virtual machines a single physical machine (PM) can host without exceeding defined performance degradation thresholds, such as 7% or 20% loss.

Test Methodology

The team tested three VM flavors (2, 4, and 8 vCPUs) on two types of Intel Xeon Silver processors: Sapphire Rapids (4410Y) and Cascade Lake (4214). They varied VM CPU usage from 10% to 100% in 10% increments and increased the VM count from 1 to 60. Benchmarks were run using CoreMark and stress-ng to measure performance and collect CPU performance counters.

Performance Degradation Factors

Analysis revealed that performance degradation follows a 1/N pattern rather than a linear decline. Two main factors drive this:

• Instructions per Cycle (IPC): As VM count increases, IPC drops due to rising Cache Misses, even before CPU cycles saturate.
• CPU Cycles: Once the PM's physical cores are saturated, available CPU cycles per VM decrease, further reducing performance.

Turbo Boost was found to increase CPU cycles by ~35% while slightly reducing IPC, resulting in a net performance gain. Additionally, vendor differences were observed; one hardware vendor showed Cache Miss rates spiking to 60%, compared to 40% for another, despite using identical CPU models.

Practical Implications

The study provides a framework for capacity planning. For example, on a Cascade Lake PM with 4 vCPU VMs at 40% average CPU usage, tolerating up to 20% performance loss allows for approximately 21 VMs per host. The authors note that while these results are specific to CPU constraints, the methodology can be extended to memory and disk resources for holistic infrastructure optimization.