AI BriefingNew Arm NEON kernel speeds up llama.cpp prompt processing by up to 6.5x
·2026.11.13 19:28
Key point
A 164-line patch adds a missing Arm NEON kernel for llama.cpp's tiled K-quant matmul, achieving up to 6.5x faster prompt processing on Neoverse N1 without affecting generation speed.
Details
- A new Arm NEON kernel using the SDOT instruction resolves a performance gap in llama.cpp v0.9.0, where low-bit quantized models (Q3_K, Q2_K, IQ family) previously fell back to slow unpacking paths on Arm CPUs.
- Benchmarks on Qwen2.5 1.5B with 512-token prompts show prompt processing speeds increasing from 11.5 to 74.5 tokens per second (6.5x) on Arm Neoverse N1, with gains of 4.9x on V1 and 4.3x on V2 for IQ3_M format.
- The patch consists of 164 added lines in two files, implementing debias-on-load and prompt-side interleaving to enable exact integer arithmetic and efficient accumulation via SDOT instructions.
- Generation speed remains unchanged, and formats with existing Arm kernels (Q4_K_M, Q5_K_M, Q6_K, Q8_0) show no performance difference, confirming the fix targets only the missing low-bit routines.
- The repository includes the patch, benchmark scripts, and raw llama-bench outputs, but has not been submitted to upstream llama.cpp due to contribution rules regarding AI-generated code.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Unsloth Releases Dynamic V3.1 Quantizations for Qwen3.8-27B and New 1-bit Data Types for Qwen3.8-2.4THuawei releases MACaQ, a model-agnostic quantizer for local inference