MoE 54% speedup
Key point
Boosted Qwen3.6-35B-A3B speed by 54% on an RTX 5070 Ti using --n-cpu-moe.
Details
This is the result of tuning Qwen3.6-35B-A3B GGUF UD-Q4_K_M (22.1GB) on an RTX 5070 Ti (16GB), Ryzen 9800X3D, 32GB RAM, llama.cpp b8829 (CUDA 13.1, Windows x64).
The key is using --n-cpu-moe 20 instead of --cpu-moe.
--cpu-moe: offloads all MoE experts to CPU, using only 3.5GB of VRAM--n-cpu-moe 20: with 40 layers total, only the first 20 layers go to CPU while the rest are loaded onto GPU, utilizing up to 12.7GB of VRAM--n-cpu-moe 20 + -np 1 + 128K ctx: generation 79.3 t/s, prompt 135.8 t/s, VRAM 13.2GB
Based on 300-token generation, --n-cpu-moe 20 reached 78.7 t/s, a 54% improvement over --cpu-moe's 51.2 t/s, and prompt speed also rose from 87.9 t/s to 100.6 t/s.
The 128K context was added at essentially no cost by reducing recurrent-state memory with -np 1, and the example run includes -ngl 99, -fa on, -ctk q8_0, -ctv q8_0, --reasoning-budget -1, etc.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.