VRAM Expert Cache

·2026.04.15 12:20

Key point

A dynamic expert cache in llama.cpp boosted Qwen3.5-122B-A10B token generation speed by 27%.

Results were shared from applying a dynamic expert cache to llama.cpp to optimize CPU+GPU token generation for Qwen3.5-122B-A10B.

Compared to the existing layer-based single-GPU partial offload approach, the dynamic expert cache showed 27% faster performance.

Key points are as follows.

Keeps frequently used experts in VRAM, reducing unnecessary movement
Improves token generation efficiency in CPU+GPU hybrid inference
Presents an alternative MoE model offloading strategy in the llama.cpp environment

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.