AI Briefing
KOSign in

llama.cpp PR #31188 fuses MoE gate/up matmuls in CUDA for up to 17.2% speedup

·2026.10.28 22:40

Key point

The optimization achieves 5.7% to 6.1% gains on RTX 5090 for Qwen3 and Qwen3.6 models.

Details

  • Pull Request #31188 in ggml-org/llama.cpp fuses MoE gate/up matmuls and GLU operations in the CUDA backend.
  • Prompt-processing speed improves by 17.2% on DGX Spark (GB10) for Qwen3-30B-A3B Q4_K_M, reaching 3410.8 tokens per second from 2911.4.
  • Qwen3.6-35B-A3B Q4_K_M sees a 9.9% increase on DGX Spark, rising from 3235.8 to 3556.7 tokens per second.
  • RTX 5090 performance gains range from 5.7% to 6.1% across Qwen3-30B-A3B and Qwen3.6-35B-A3B variants, with Q4_K_M and UD-Q4_K_M quantizations tested.
gpucudaqwenmoellama.cppinference-optimization
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

RedditOpenBMB MiniCPM-V 4.7 Achieves 20.51 tok/s on iPhone 18 Pro Max via Lookahead Expert SchedulingRedditllama.cpp PR #29353 introduces GDN chunked kernel for CUDA and HIP, boosting Qwen inference speed by up to 20.6%PyTorchKRLafa Achieves 2.54x Speedup with Flow-Based GenerationHacker NewsCohere Reports Qwen3.8-Flash-Next Attempting Unauthorized Network Uploads in SandboxKT CloudKT Cloud Aims to Secure 1GW AI Data Center Supply Capacity by 2031PyTorchKRStepFun Step 5 Preview Weights Released, Runnable on 8 H200s