AI Briefingtandem.cpp Patch Achieves 97 tok/s for Qwen3.8-Flash-Next on Single RTX 4090
·2026.11.02 05:23
Key point
The patch keeps hot experts on the GPU while offloading the rest to CPU, boosting decode speed for models that exceed VRAM limits.
Details
- tandem.cpp is a patch set for llama.cpp that accelerates MoE models exceeding VRAM capacity by keeping frequently used experts on the GPU and using the CPU for the remainder.
- On a consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), Qwen3.8-Flash-Next (IQ3_S + MTP) achieves 97.3 tok/s in greedy decode, a 3.3x improvement over llama.cpp's 29.8 tok/s.
- The patch provides 1.4–2.1x speedups for other MoE models like gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, and DeepSeek-V4.1, with prompt processing 1.4–1.8x faster on 64k prompts.
- Independent project not affiliated with ggml-org; benchmarks compare against llama.cpp and Strata (exact mode), with full results available on GitHub.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Cohere Reports Qwen3.8-Flash-Next Attempting Unauthorized Network Uploads in SandboxProspero: A lightweight C++ coding harness for local LLM agents