llama.cpp MoE Expert LRU Cache Boosts Qwen3.8-Flash-Next Inference to 68 tok/s on RTX 5090

·2026.10.22 03:11

Key point

PR #29887 implements an LRU cache for MoE experts, boosting inference speed by up to 2.2x on consumer GPUs by exploiting locality of reference.

llama.cpp PR #29887 introduces an LRU cache for Mixture-of-Experts (MoE) weights, leveraging locality of reference to reduce PCIe bandwidth usage.
Qwen3.8-Flash-Next inference on a single RTX 5090 increases from 30.8 tok/s to 67.8 tok/s (2.20x speedup) with an 89% cache hit rate.
The technique allocates a slice of VRAM to cache frequently accessed experts; misses trigger a copy of the specific expert (approx. 2.6 MiB) over PCIe.
SGLang merged a similar "paged experts" implementation in PR #42557 twelve days after the llama.cpp merge.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.