llama.cpp PR #29353 introduces GDN chunked kernel for CUDA and HIP, boosting Qwen inference speed by up to 20.6%

Pull Request #29353 to the llama.cpp repository introduces a GDN chunked kernel for both CUDA and HIP backends, resulting in measurable inference speed improvements for Qwen models.

Performance Gains by Hardware

The implementation demonstrates significant throughput increases, particularly on newer NVIDIA hardware:

• RTX PRO 6000 Blackwell: Achieved the highest gains, with Qwen3.5-35B-A3B-Q8_0 seeing a +20.6% increase at 2048 prompt tokens (13,653.58 to 16,459.56 tok/s) and +17.8% at 4096 tokens.
• RTX 5090: Showed strong improvements, with Qwen3.5-35B-A3B-Q4_K_S gaining +16.4% at 2048 tokens and +17.9% at 4096 tokens.
• RTX 4090: Delivered moderate gains, such as +10.8% for Qwen3.5-35B-A3B-Q4_K_S at 2048 tokens.
• Spark GB10: Provided consistent but smaller improvements, ranging from +3.4% to +5.7% across tested Qwen variants like Qwen3.6-35B-A3B-UD-Q4_K_M and Qwen3.8-Flash-Next-UD-IQ4_XS.
• RTX 3090: Showed the most modest gains, with Qwen3.5-35B-A3B-Q4_K_S improving by +6.9% at 2048 tokens, while Qwen3.8-27B-Q4_K_M saw only +0.9% to +2.2%.

Model-Specific Observations

The performance benefits vary by model architecture and quantization:

• Qwen3.5-35B-A3B models consistently showed the largest relative speedups across all tested GPUs.
• Qwen3.8-27B models exhibited smaller gains, typically between +0.9% and +6.4%.
• The kernel supports various quantization formats including Q8_0, Q4_K_S, Q4_K_M, UD-Q4_K_M, UD-Q8_K_L, and UD-IQ4_XS.