llama-cpp-turboquant Adds ConvRot Quantization Support

·2026.08.24 10:16

Key point

The addition of ConvRot quantization to llama-cpp-turboquant brings the quality of Q6 quantized models close to Q8 levels.

Details

The ConvRot quantization method has been officially integrated into the llama-cpp-turboquant project. Applying this method improves the quality of Q6 quantized models such that their KLD/PPL metrics nearly reach the level of Q8 quantization.

Additionally, the Q6_CR and Q5_CR versions show slight performance improvements compared to the existing Q6 and Q5 quantizations. The --moe-cache auto option, useful for running MoE models larger than VRAM capacity, is also provided.

This update is expected to help recover quality lost in turbo4/3/2 quantization, and the initial decoding and crash issues have now been resolved.

open-source quantization llama-cpp llm-inference convrot

Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.