ik_llama.cpp PR adds AVX2/VNNI acceleration for Q2_0_R8 quantization

·2026.11.01 03:06

Key point

The optimization targets low-bit models from ISTA-DASLab, offering two workflows for dynamic or offline repacking of GGUF files.

PR #2785 in ik_llama.cpp introduces Q2_0_R8 support with dedicated AVX2/AVX-VNNI kernels for GSQ-RCO and ternary quantized models.
Benchmarks on an Intel Core i5-13420H show a ~45% increase in prompt processing speed and ~22% increase in token generation compared to native Q2_0.
Perplexity remains stable at 13.0763 for Q2_0_R8, matching the runtime repacked version and slightly differing from native Q2_0 (13.0721).
Users can apply the optimization via the -rtr flag for runtime repacking or by converting models offline using llama-quantize --repack.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.