Toss Securities Improves Search Throughput by 41% with Milvus NUMA Placement Optimization

The Data Infra team at Toss Securities disclosed the process of estimating the required server resources before operating a Milvus-based vector search service in an on-premises environment. Initially, they used the HNSW index, but due to excessive memory usage and failure to meet the target throughput (QPS), they switched to HNSW_SQ8. However, they discovered an issue where throughput increases were limited despite high CPU usage, and proceeded with optimization by analyzing the NUMA (Non-Uniform Memory Access) structure.

NUMA Placement Optimization

Experimental results showed that approximately 50% of DRAM reads during search occurred from the opposite socket (Remote Access), causing long CPU wait times for data. To resolve this, they utilized Kubernetes' CPU Manager, Memory Manager, and Topology Manager to place the CPU and memory of QueryNodes on the same NUMA node.

• Remote DRAM ratio: Decreased from approximately 50% to 0.8%
• IPC (Instructions Per Cycle): Improved from 0.37 to 0.55
• Throughput (QPS): Increased by 41% from 405 to 572 at ef48
• Latency (p99): Reduced by 48% from 3.94 seconds to 2.04 seconds

Establishing Sizing Criteria

Based on the optimized configuration, they measured per-node throughput according to vector dimension, count, and search settings (ef value). Through this, they developed a simulator that calculates the required number of servers when inputting the search conditions required by new services. When the vector dimension doubled, throughput tended to decrease to approximately 1/3, and server sizing is estimated by reflecting the more restrictive constraint between memory capacity and QPS.