SpecMD: A Comprehensive Study on MoE Expert Prefetching and Caching

Mixture-of-Experts (MoE) models use only a subset of experts per inference thanks to sparse expert activation, but achieving real-world performance requires expert caching and prefetching. Existing cache policies have not sufficiently accounted for hardware-specific differences and the interactions between policies.

SpecMD is a benchmark framework that standardizes and compares ad-hoc cache policies across diverse hardware configurations. Under controlled conditions and realistic constraints, it reproduces and extends existing MoE caching strategies to consistently measure the performance of each policy.

Key experimental results are as follows.

• MoE expert access did not align well with the temporal locality assumed by LRU and LFU.
• Based on this finding, Least-Stale was proposed, reducing collision misses by up to 85x compared to LRU.
• On OLMoE, using only 5% VRAM cache, or 0.6GB, achieved a hit rate above 88% and up to a 34.7% reduction in TTFT (Time-to-first-token).