AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
Hipfire, a new inference engine optimized across AMD GPUs, along with the mq4 quantization method, has been released.
A decensored version of Qwen3.6 35B A3B using Heretic has been released.
In an 8GB VRAM / 32GB RAM environment, Q4_K_XL was slightly faster than Q4_K_M and also produced fewer output tokens.
On a 3070 8GB setup, Qwen3.6-35B ran faster with larger quants.
The quantization levels of MLX Q and oQ were compared using KL divergence and RAM usage.
Max-and-Omnis released GGUF files for Nemotron-3-Super-64B-A12B-Math-REAP.
Qwen3.6 27B IQ4_XS recorded 22 tok/s on an RTX 5060 Ti 16GB.
Qwen3.6-35B-A3B ran with 64K context on an RTX 5070 12GB.
A 4-bit GPTQ-Pro quantized model has been released to run Qwen3.6-27B on Ampere GPUs.
Qwen3.6-27B is a 27B dense model that claims coding performance surpassing the previous ultra-large MoE flagship.
An uncensored Aggressive version of Qwen3.6-27B and K_P quantizations have been released.
A Metal kernel implementation combining INT3 model compression with INT2 KV cache has been released.
Using Hadamard rotation and a Lloyd-Max codebook, the PLI of Gemma 4 E2B was compressed to 2 bits.
On-device streaming ASR achieved **8.20% WER** and **0.56s** latency on CPU.
A 4-bit reasoning model based on DeepSeek-R1-32B has been released, reduced to 20GB.
Even with 32GB VRAM, llama.cpp's `--fit` ran Qwen3.6 Q8 at a 256k context at 57 t/s.
Running the same prompt across 9 local models revealed a correlation between quantization provider differences and code length.
By optimizing the q1_0/q8_0 dot path, up to 67x throughput improvement was reported on x86.
With 1.58-bit ternary weights, an 8B model scored 75.5 points.
A benchmark of 4 tasks and 9 datasets verified the limits of LLMs' context understanding.
In Terminal-Bench 2.0, MiniMax M2.7 scored 41.3%, lower than M2.5.
A GGUF Q4_X conversion of Kimi-K2.6 has been uploaded to Hugging Face.