AI Briefing
KO

llama.cpp adds flash-attn support

·2026.04.29 06:48

Key point

llama.cpp has added DKQ=320/DV=256 flash-attn CUDA support for Mistral Small 4.

Details

A flash-attn kernel for DKQ=320 / DV=256 has been added to ggml-cuda. The key point is limiting to ncols2=32, moving the CPU fallback that occurred with Mistral Small 4's GQA=32 combination onto the CUDA path.

The patch includes MMA-f16 and tile kernel config, dispatch logic, template instances, and a tile .cu file for Mistral Small 4.

Benchmarks were measured on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition.

  • Before the change: pp512 179.64±2.66, tg32 33.05±0.30
  • ncols2=32: pp512 3752.47±26.33, tg32 185.92±4.43
  • ncols2=8: pp512 3678.35±21.68, tg32 182.24±4.00

In a follow-up fix, a bug where the per-warp-group base index was missing at ncols=32, causing results to diverge from the CPU output, was fixed, and a FLASH_ATTN_EXT failure involving sinks=1 was also resolved. In the latest retest, pp512 3655.55±15.81 and tg32 186.31±4.23 were confirmed.

In multi-GPU environments as well, pp512 improved from 348.35 → 1310.13 and tg128 from 59.12 → 93.06.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.