AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
The new STQ1_0 format achieves 1.3125 bits per weight, reducing the model's storage footprint from roughly 1.5TB while maintaining performance comparable to IQ1_M.
New NVFP4 and GGUF checkpoints use quantization-aware distillation to achieve lower KL divergence than reference 4-bit models from NVIDIA, Google, and community quantizers.
The release includes quantized versions of the 32B, 7B, 3.7B, and 0.9B parameter models.
The new variants achieve a 6.58% increase in SWE performance and use 20% fewer reasoning tokens while compressing the KV cache by 4x.
The 74B parameter MoE model is packed into 24GB using 2-bit quantization, with a 4GB variant releasing Oct. 23.
The new model achieves 13+ tokens/s decoding speed on iPhone 17 Pro Max using Lookahead Expert Scheduling.
The new model achieves 85.9% of the BF16 baseline score while being 13.9× smaller in file size.
The model retains 98.2% of FP16 intelligence while reducing size to ~5.9 GB for llama.cpp.
The official implementation supports compression down to 0.1 bits per weight across models including Llama 3, Phi-4, and Qwen3.
New mixed-precision 3-bit weights for Qwen3.8 27B enable a 569,878-token reusable cache on a single AMD R9700, accelerating agent workflows by up to 12x for first-token latency with a 2.9-point MMLU-Pro drop vs MXFP4.
Victoria achieves 70.0% on Terminal-Bench 2.1 using REAP pruning, while Maple improves Canadian regulatory accuracy from 6.6% to 21.8%.
Hugging Face Transformers has released a feature that allows direct inference of GGUF models in a compressed state without dequantization on Apple Silicon Macs, utilizing llama.cpp's ggml Metal kernels.
The new TQ method achieves a mean KLD of 0.0282 and 92.4% top-1 accuracy on Qwen 3.8 27B at 4-bit without requiring calibration data.
The new Swift 1.5 27B model reduces thinking tokens by 58.5% while improving accuracy, with Flash Next offering a 1.8x speed increase.
Hugging Face has added native support for the GGUF format to the Transformers library, enabling debugging and flexible use of locally quantized models.
GGUF quantization files for the K2-Horizon model family have been released, but a forked version of llama.cpp is currently required.
TextCLF has released 'Quant Factory', an open-source tool that enables immediate 4-bit quantization without calibration data.
The 'Gewell' inference engine, featuring lossless KV cache reduction for the Gemma 4 31B model, has been released, improving VRAM efficiency and throughput in Blackwell GPU environments.
PrismML has released 'Ternary Bonsai 2 27B', a 27B multimodal LLM quantized to ternary values that maintains 98.2% of FP16 performance.
A single-file C-based inference engine has been released that performs text and image inference for Gemma 1/2/3 without external dependencies.
As AI inference demand surges, new hardware architectures and quantization techniques that separate workloads into Prefill and Decode stages and resolve memory bottlenecks are emerging.
Voodoo Dynamic Quant, a tool that selects the optimal quantization level for each tensor using gradient descent, has been released under the MIT license.
Pinterest reduced serving costs by up to 30% by introducing SSD serving and quantization technologies to the Manas platform.
Edge0 released an inference framework that runs 35B MoE models on 3GiB of memory using SSD expert streaming technology.