AI Briefing#gguf
The latest AI and developer news about #gguf, with the original source and a short summary.
The latest AI and developer news about #gguf, with the original source and a short summary.
The release uses an LLM to natively edit weights, outperforming Unsloth and EXL3 in KL divergence and top-1 accuracy at 128K context.
The release accompanies the MACaQ paper (arXiv:2611.10649) and is available via the huawei-csl GitHub repository.
New 2-bit to 6-bit quantizations achieve higher accuracy on agentic and coding benchmarks using 256K and 128K context windows.
The update adds a --shard argument to combine model fragments from different locations and supports loading PLE files from separate GGUFs for Gemma4, Qwen3.8Next, and GLM5.3Flash.
The new quantization method reduces token usage by up to 50% and improves accuracy by 15% compared to other quants of the same size.
New quantization files improve accuracy by 10-15% on agentic benchmarks and introduce a new engram switching process for multiple precisions.
Unsloth reports its Dynamic 3.1 GGUFs for DeepSeek-V4.1 outperform GSQ-RCO by 10-20% on agentic tasks at 512K context, while statistical analysis suggests GSQ-RCO's claimed 99.80% recovery for IQ2_XXS is nearly impossible.
The expert-pruned build retains 98.7% of LiveCodeBench v6 performance while reducing transformer weights to 72.1 GB.
The 1.58-bit MoE model achieves 172 tok/s decode and 7000+ tok/s prefill speeds in both Stock llama.cpp and the PrismML fork.
ISTA DASLab releases GSQ-RCO quantized GGUFs for the 763.2B parameter DeepSeek-V4.1-Flash-1017, achieving 100% benchmark retention at 2.00 bpw on a 12GB GPU.
The new quantization offers three GGUF sizes ranging from 10.93 to 20.82 GB on Hugging Face.
The release includes quantized versions of the 32B, 7B, 3.7B, and 0.9B parameter models.
LiquidAI has released LFM2.5-Encoder-350M and LFM2.5-Encoder-230M, with the 350M variant described as a multilingual bidirectional encoder supporting 15 languages and on-device inference.
The new quantized models achieve near-BF16 performance at 3.50 bpw, while the expert-pruned Coder build reduces the resident working set to 29.6 GB.
Hugging Face Transformers has released a feature that allows direct inference of GGUF models in a compressed state without dequantization on Apple Silicon Macs, utilizing llama.cpp's ggml Metal kernels.
Hugging Face has added native support for the GGUF format to the Transformers library, enabling debugging and flexible use of locally quantized models.
GGUF quantization files for the K2-Horizon model family have been released, but a forked version of llama.cpp is currently required.
The open-source dynamic quantization tool 'Dynamic Quantiser' has been released, offering higher accuracy than standard quantization in VRAM-constrained environments for llama.cpp-compatible GGUF files.
Hugging Face Transformers now directly supports the GGUF format from llama.cpp, simplifying local inference on Apple Silicon.
Voodoo Dynamic Quant, a tool that selects the optimal quantization level for each tensor using gradient descent, has been released under the MIT license.
GGUF model uploader Bartowski released research on performance improvements for quantized models using new tensor layout maps.
IST-DASLab released a Qwen3.8-27B GGUF model with significantly improved performance over previous versions by applying GSQ and RCO technologies.
Numerous cases have been discovered where GGUF filenames differ from the actual saved bit counts due to quantization logic issues in llama.cpp.
Unsloth released the Qwen3.8-27B Dynamic 3.0 quantized model, improving accuracy.