LLM Serving: The Difference Between Launching and Launching Well

The LLM serving ecosystem is evolving rapidly, and 'launching well' has become more important than simply deploying models. Toss Securities built the MLHub platform to normalize logs and metrics, and systematized server monitoring using Grafana and Kibana.

Unlike existing ML models, LLMs generate tokens sequentially, so monitoring metrics shifted to Tokens Per Second and TTFT (Time To First Token). This approach resolved three key issues.

• Prefix Cache Not Enabled: They discovered that the Prefix Cache Hit Rate was 0% for certain models. Enabling it reduced TTFT by approximately 1/10.
• vLLM Bug: They identified an issue where finish_reason ended with length for reasoning models, confirmed it as a vLLM bug, and applied a workaround to lower the error rate.
• Throughput Optimization: By identifying low KV Cache usage, they tested increasing concurrency up to 20, which boosted traffic throughput by 4x without adding resources.

These experiences demonstrate that centralized monitoring, metric observation, hypothesis validation, and a rapid rollback system are core to operating LLM services.