Experience Migrating an ML GPU Model Server to a CPU Server While Maintaining Performance

With limited GPU resources, this is a case study of migrating a relatively small ML model server that was previously running on GPU to CPU serving while maintaining service quality. The goal was not simply cost reduction, but converting the GPU server to a CPU server without any performance drop, achieving an annual cost savings of about 400 million won.

In the initial comparison, the image scoring model deployed on CPU showed 10x lower rps and 10x slower response time compared to GPU. Since additional scale out was not being considered, roughly a 10-20x level of performance improvement was needed.

Initially, they tried increasing TorchServe worker count to boost throughput, but this actually caused performance to degrade. The cause was that torch thread's default value is set to the number of physical cores, so increasing workers led to logical thread contention and a front end bound occurring in GEMM operations.

The solution had two steps:

• Increased the worker count, but adjusted each worker's torch thread to match the pod's CPU limit and physical core count.
• On top of this, applied ipex_enable=true and cpu_launcher_enable=true to pin threads at the socket level.

As a result, rps rose to about 4x the initial level (roughly 7.9-8 rps). However, since ipex is particularly effective for nn inference, a greater improvement (2.5 rps → 17.5 rps, about 7x) was confirmed in the food recognizer, which is a pure nn model, compared to the image scorer, which has an svr attached after inference.

The latency issue was approached separately through model lightweighting. Since accuracy was important for the service, pruning was excluded, and instead approaches like Knowledge Distillation (KD) were used to make the model itself lighter, aiming to reduce CPU inference time. They also confirmed a version issue where ipex-based multi-worker pinning only works properly on torchserve 0.6.1 and above, requiring them to dig directly into the code for older versions.