AI BriefingELDR Routing Technique Reduces MoE Serving Latency
Key point
ELDR routing, which accounts for expert locality in MoE models, cuts decode latency by up to 13.9%.
Details
For efficient inference of MoE (Mixture-of-Experts) models in Prefill-Decode (PD) disaggregated LLM serving environments, the ELDR (Expert-Locality-Aware Decode Router) technique has been proposed.
Existing decode routers simply consider load balancing across Workers, but MoE models have the characteristic that latency varies depending on whether the weights of the Experts activated at each decode step are already loaded. ELDR addresses this through the following mechanisms.
- Expert Signature: Based on the expert activation pattern that occurs during the Prefill stage of a request, it generates a signature that predicts which experts will be activated during generation.
- Locality-Band Routing: Based on the signature space partitioned offline via K-means clustering, requests are assigned to the least loaded Worker among those holding the relevant experts.
- Signature Cache: Through a signature cache linked with the KV cache, accurate routing is maintained even in Prefix Caching environments.
Implemented in vLLM and tested at a scale of 40 GPUs, it showed performance improvements of reducing median TPOT (Time Per Output Token) by 5.9% to 13.9% compared to existing state-of-the-art load balancing baselines.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
vLLM Publishes Performance Benchmarks for 5 Speculative Decoding Methods on AMD GPUs
GLM-5.3-Flash Builds Inference Infrastructure on Cluster of Over 100,000 Chinese Accelerators, Achieving ~3x Performance Boost