Request Queuing Strategies for LLM Performance Optimization

Inference engines like vLLM or HuggingFace TGI use a single queue by default. This causes a problem where, if a specific 'heavy user' sends a large volume of requests, the queue becomes occupied, blocking other users' requests from being processed.

To solve this, we propose a structure that places a separate API Server (LLM-Server) in front of the inference engine. By creating an independent queue per user and scheduling requests via a Round-robin method, Fair Scheduling becomes possible, ensuring fairness before requests are passed to the inference engine.

Furthermore, the following optimization approaches can be considered:

• Processing time-based priority: A method that prioritizes shorter requests by considering the generation length of the request
• KV-cache-aware routing: A method that maximizes cache hit rate by batching based on similarity between requests (e.g., NVIDIA Dynamo, AIBrix)
• Utilizing priority queues: A method that assigns high priority to interactive services such as real-time chat, and low priority to batch tasks such as bulk code review