AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
A method for optimizing Qwen3.6 inference performance by grafting an MTP layer onto an Unsloth quantized model is introduced.
ICLR 2026's ParoQuant has released a 4-bit quantization method along with code and models for reasoning LLMs.
vLLM has resolved the unimplemented TurboQuant error affecting Qwen 3.5+ models caused by an issue with Mamba layers.
APEX, a mixed-precision quantization strategy for MoE models, has expanded to over 30 model types and launched a new ultra-compressed tier.
A reproducible setup was shared for running Qwen 3.6 35B MoE on an RTX 3090 with 262K context.
Ran Qwen3.6-35B-A3B at 23 t/s on a 5-year-old laptop with 6GB VRAM.
MiniMax M2.7 AWQ-4bit was compared for performance and energy efficiency on 2x Spark and 2x RTX 6000 96GB.
Unsloth released a fixed version of the Mistral Medium 3.5 128B GGUF model that had been producing errors in long contexts.
In Tesla P40 benchmarks, the saw-family KV quantization was the most stable.
RedHatAI released the Qwen3.6-27B FP8 model on Hugging Face.
Ran a Qwen3.6-27B UD-Q5_K_XL benchmark on an RTX 5090 with 19 runs and 93.9k tokens.
NVIDIA released an NVFP4 quantized checkpoint of the Gemma 4 26B model.
An MLX 4-bit conversion of Mistral Medium 3.5 128B has been released.
A 3-bit vector search package that runs in browsers and Node.js has been released.
GGUF quantization files and recommended variants for IBM Granite 4.1 30B have been released.
On Qwen3.6-35B-A3B, KLD and memory usage were compared across oQ, Q, MXFP, and UD MLX.
A TurboQuant inference path was added to Runtime Valkyr, achieving 120 tok/s on Gemma with an RTX 3090.
Native NVFP4 in llama.cpp b8967 boosted prefill speed by up to 68%, but generation speed stayed nearly the same.
llama.cpp b8967 adds native NVFP4 support for Blackwell.
Measured Qwen 3.6 KV cache performance differences by format from 0 to 1M context on M5 Max.
Compared the performance of Qwen 3.6 27B's BF16, Q4_K_M, and Q8_0 quantizations.
Qwen3.6 27B and 35B-A3B were compared across multiple engines on 2x 5060 Ti 16GB.
It explains the principles of vector quantization in TurboQuant and its random rotation-based compression method.
Qwen3.6-27B was converted into a mixed 3-bit/5-bit quantization for MLX.