AI BriefingKQuant Quantization Technique Reduces KL Divergence by Up to 3.7x for Qwen Models on MLX
·2026.11.12 19:47
Key point
Open-source KQuant packs for Qwen models on Hugging Face achieve up to 3.7x lower KL divergence than public 4-bit baselines, with 6-15% faster inference on M5 Pro for 4B, 9B, and 27B models.
Details
- KQuant is a new quantization technique released by Nace, offering open-source MLX packs for Qwen3.5-4B, 9B, Qwen3.8-27B, Qwen3.6-35B-A3B, and Qwen3.8-Flash-Next on Hugging Face.
- Measured against public 4-bit baselines (mlx-community or lmstudio-community), KQuant Q4 achieves lower KL divergence, with the largest improvement of 3.7x for Qwen3.8-Flash-Next (0.067 vs 0.249) and 2.6x for Qwen3.5-9B (0.030 vs 0.081).
- The technique uses real text to determine scales and offsets for weight groups and allocates extra bits to sensitive layers, with search performed on cloud GPU clusters before release.
- Inference speed on M5 Pro is 6-15% faster than public 4-bit packs for 4B, 9B, and 27B models when using Kapitsa 0.2.35 with the KFlash drafter; 35B is compared against Youssofal MTPLX 4-bit. No custom kernels or special loaders are required.
- Q5 variants are also available for 9B, 27B, and 35B models, offering further precision improvements at slightly larger file sizes.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
New Arm NEON kernel speeds up llama.cpp prompt processing by up to 6.5xHuawei releases MACaQ, a model-agnostic quantizer for local inference