MoE flipped the script

We tested --cpu-moe hybrid offloading on a setup with dual RTX 5060 Ti (total 32GB VRAM) and 64GB system RAM.

Three configurations were compared back-to-back using the same bench harness, with MoE running on ghcr.io/ggml-org/llama.cpp:server-cuda13 and dense running on a TurboQuant build. The settings were 10 sequential runs, a 4 concurrent workers stress test, 128/256 max tokens, 2 warmup, and a 5-minute measurement.

• Qwen 3.5-27B dense: generation 18.3 tok/s, P50 7,196 ms, stress 10.4 tok/s, 52 req/5min
• Qwen 3-Coder-30B-A3B: generation 31.1 tok/s, P50 2,286 ms, stress 12.0 tok/s, 113 req/5min
• Qwen 3.6-35B-A3B: generation 21.7 tok/s, P50 6,160 ms, stress 6.8 tok/s, 38 req/5min

The key finding is that --cpu-moe was far more efficient than dense 3.5, but Qwen 3.6 was slower than the Coder variant due to its larger total parameter count. On the other hand, its quality metrics were stronger. Citing SWE-bench Verified 73.4% vs 50.3% and Terminal-Bench 2.0 +11 points, the author recommends 3.6 for agentic, multi-step tasks and Coder for fast code completion.

Additionally, prompt processing speed was fastest on dense at 160 tok/s, while hybrid runs stayed in the 30-95 tok/s range, leading to the conclusion that dense is better suited for long-form RAG or loading large prompts.