llama.cpp PR #31533 adds CUDA MMQ kernel for 2-24 column batches, boosting speculative decoding speed by up to 34.9%

·2026.11.12 23:10

Key point

New CUDA mat-mul kernel for 2-24 src1 columns improves speculative decoding performance by 10.1% to 34.9% across tested NVIDIA GPUs.

New CUDA mat-mul kernel in llama.cpp PR #31533 targets 2-24 src1 columns, addressing inefficiencies in existing MMVQ and MMQ kernels for small batches and speculative decoding.
The kernel reads each weight once and decodes it into int8 MMA fragments to multiply with all columns on tensor cores, avoiding repeated multiply work and block-wide barriers.
Performance gains on Qwen3.8-27B UD-Q4_K_M with DFlash2 draft (7 tokens) range from 1.106x (10.1%) on DGX Spark to 1.344x (34.9%) on RTX 3090 Ti.
Supported quantization types include Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL, and IQ4_XS on NVIDIA GPUs from Ampere to DGX Spark.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.