Toss Securities Improves Milvus Search Throughput by 41% via NUMA Placement Optimization

Before operating a Milvus-based vector search service in an on-premises environment, the Toss Securities Data Infra team conducted experiments to accurately estimate the required server resources. Initially, it was difficult to predict actual workload resource usage based solely on public benchmark QPS, and they needed to find an optimal configuration that simultaneously satisfied search quality (recall) and throughput (QPS).

NUMA Architecture and CPU Latency Analysis

The experiments revealed a phenomenon where throughput stagnated despite high CPU utilization. Measuring IPC (Instructions Per Cycle) showed a low value of 0.37, indicating significant time spent by the CPU waiting for data. Due to the irregular memory access patterns inherent in HNSW indexes, the remote DRAM access ratio—reading memory from the opposite socket in a NUMA (Non-Uniform Memory Access) architecture—reached approximately 50%.

NUMA Placement Optimization Results

By leveraging Kubernetes' CPU Manager, Memory Manager, and Topology Manager, the team placed the CPU and memory of QueryNodes on the same NUMA node. As a result, the remote DRAM access ratio plummeted to 0.8%, and IPC rose to 0.55. Under the same hardware resources and recall@10 0.954 conditions, QPS increased by 41%, and p99 latency was reduced by 48%.

Establishing Sizing Criteria

Based on the optimized configuration, they measured changes in throughput according to vector dimensions and search settings (ef values). They confirmed that doubling the dimension reduces throughput to approximately one-third, and developed a simulator to calculate the number of servers reflecting this trend. This enables the accurate pre-estimation of server scale required for new service introductions.