AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
Unsloth updated Qwen3.8-27B GGUFs with Dynamic V3.1 for 5-10% accuracy gains and introduced new 1-bit quantization types for the 2.4T parameter model.
The release uses an LLM to natively edit weights, outperforming Unsloth and EXL3 in KL divergence and top-1 accuracy at 128K context.
The framework supports weight and activation quantization for LLMs, text-to-image, and text-to-video models while maintaining inference speed.
A 164-line patch adds a missing Arm NEON kernel for llama.cpp's tiled K-quant matmul, achieving up to 6.5x faster prompt processing on Neoverse N1 without affecting generation speed.
Precision Numerics releases Boole-32B-A2.5B-Preview, a 1.8-bit MoE model that is 9x smaller than BF16 and 2x faster than INT4.
New CUDA mat-mul kernel for 2-24 src1 columns improves speculative decoding performance by 10.1% to 34.9% across tested NVIDIA GPUs.
Open-source KQuant packs for Qwen models on Hugging Face achieve up to 3.7x lower KL divergence than public 4-bit baselines, with 6-15% faster inference on M5 Pro for 4B, 9B, and 27B models.
The project replaces all floating-point operations with exact integer arithmetic, ensuring bit-identical logits across different GPUs and CPUs without requiring pinned kernel orders.
MACaQ quantizes a 122B-parameter model in 64 minutes using 32.5 GB of device memory, supporting text, image, video, and audio models across llama.cpp, vLLM, Diffusers, and PyTorch.
New 2-bit to 6-bit quantizations achieve higher accuracy on agentic and coding benchmarks using 256K and 128K context windows.
The model compresses MiniMax-H3 to 13.7B parameters for 4-8 step generation, running on consumer RTX 5090 GPUs in 92 seconds for 5.2-second clips.
The new quantization method reduces token usage by up to 50% and improves accuracy by 15% compared to other quants of the same size.
The method identifies that FP4 errors stem from low-entropy tokens and uses trace-aligned training to reach 65.94% AIME-120 accuracy on R1-Qwen-14B.
New quantization files improve accuracy by 10-15% on agentic benchmarks and introduce a new engram switching process for multiple precisions.
The team combined mixed-precision quantization, DFlash2 speculative decoding, and custom CUDA kernels to reduce latency and improve throughput on a single GPU.
Unsloth reports its Dynamic 3.1 GGUFs for DeepSeek-V4.1 outperform GSQ-RCO by 10-20% on agentic tasks at 512K context, while statistical analysis suggests GSQ-RCO's claimed 99.80% recovery for IQ2_XXS is nearly impossible.
The expert-pruned build retains 98.7% of LiveCodeBench v6 performance while reducing transformer weights to 72.1 GB.
Swift GLM-5.3-Flash achieves -61.4% thinking tokens on MMLU-PRO with -0.46% average accuracy loss across nine benchmarks.
The 1.58-bit MoE model achieves 172 tok/s decode and 7000+ tok/s prefill speeds in both Stock llama.cpp and the PrismML fork.
ISTA DASLab releases GSQ-RCO quantized GGUFs for the 763.2B parameter DeepSeek-V4.1-Flash-1017, achieving 100% benchmark retention at 2.00 bpw on a 12GB GPU.
The model runs at 72x real-time on CPU and 1120x on GPU, though it may slur difficult words or long numbers.
The new quantization offers three GGUF sizes ranging from 10.93 to 20.82 GB on Hugging Face.
A frozen 16 MB vocabulary table matches or exceeds learned embeddings on key benchmarks while reducing model size to 46 MB and boosting CPU inference speed by five to six times.
The system uses a lightweight quantized draft model to predict expert usage, reducing PCIe I/O bottlenecks on commodity hardware.