Selecting a Framework for Model Serving Optimization and Maximizing Serving Performance

Kakaopay pushed to introduce an efficient model serving framework to meet the requirements of its face recognition and OCR services. Previously, Python for model development and Kotlin/Spring Boot for serving were separated, causing consistency issues and reduced development productivity during the pre/post-processing code conversion process.

To improve inference speed, optimization engines such as ONNX and TensorRT were utilized. Test results showed performance in the order of TensorRT > ONNX > Tensorflow, but consistency could be off depending on the model structure, requiring consideration of the trade-off between speed and accuracy.

To select the optimal framework, FastAPI, Tensorflow Serving, and Triton were chosen for comparison. FastAPI was used as a baseline for directly building a server, while Tensorflow Serving and Triton were evaluated based on the variety of model formats they support and their serving performance.

The experiments were conducted based on a ResNet50+alpha model in an AWS ECS (G4dn instance) environment. Locust was used to measure requests per second (Throughput) and latency (Latency), verifying the performance and stability of each framework.