AI Briefing
KO

llama.cpp MMQ Optimization

·2026.04.25 23:22

Key point

The llama.cpp CUDA MMQ kernel improved speed in some ranges using fastdiv and dynamic tiling.

Details

This is a PR that reduces the stream-k overhead occurring in CUDA's MMQ kernel.

There are two key changes.

  • Using fastdiv, which precomputes some values on the CPU, to lower the cost of integer division.
  • Depending on conditions, using tiling instead of stream-k to skip the fixup stage. Here, the number of CUDA blocks is dynamically adjusted to match the number of tiles, so that each block processes exactly one tile.

Unlike previous proposals, this PR does not branch based on whether it's MoE, and instead chooses tiling whenever tiling's efficiency loss is 10% or less.

Benchmarks were conducted on an NVIDIA P40 using llama 8B-class quantized models. Overall, the results were not dramatic changes: in low-bit ranges there was a slight decrease or virtually no difference, while in some 3-bit ranges there was an improvement of about 1-3%. For example, in the IQ3_S series, gains of about 2-3% were reported across the microbatch 64-2048 range.

In summary, this is a fine-grained optimization patch for llama.cpp's CUDA MMQ path; the perceived effect is not large, but measurable improvements were confirmed in certain configurations.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.