AI Briefing
KOSign in

Huawei releases MACaQ, a model-agnostic quantizer for local inference

·2026.11.12 05:23

Key point

MACaQ quantizes a 122B-parameter model in 64 minutes using 32.5 GB of device memory, supporting text, image, video, and audio models across llama.cpp, vLLM, Diffusers, and PyTorch.

1 / 2

Details

  • Huawei CSL releases MACaQ, an open-source quantization tool that supports text, image, video, and audio models for local inference and serving.
  • The tool quantizes a 122B-parameter model in 64 minutes using only 32.5 GB of device memory.
  • MACaQ achieves lower perplexity than MR-GPTQ and DuQuant++, or matches their best Llama-3-8B perplexity with 11x less quantization time.
  • Backends include llama.cpp (2/3/4-bit GGUF), vLLM (W4A4), Diffusers (W4A4 for image/video/audio), and PyTorch (W4A4 for speech).
  • On Qwen3.8-27B, MACaQ W4A4 in vLLM achieves 2.33x faster prefill and 2.58x faster decode compared to BF16, with a perplexity increase from 2.8463 to 2.8994.
  • The project is licensed under Apache 2.0 and includes a paper on arXiv (2611.10649).
open-sourcequantizationlocal-inferencellama.cppvllmhuawei
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

RedditMACaQ: Open-source quantizer with custom vLLM kernels and lower low-bit KLDPyTorchKRllama.cpp v1.0.0, First Major Release in Three and a Half YearsHacker NewsUnsloth Releases Dynamic V3.1 Quantizations for Qwen3.8-27B and New 1-bit Data Types for Qwen3.8-2.4THuggingFace Blogllama.cpp and ggml reach v1.0.0 with stable API and ABI guaranteesHacker NewsNew Arm NEON kernel speeds up llama.cpp prompt processing by up to 6.5xRedditllama.cpp merges dedicated Vulkan Flash Attention prefill kernel for Intel Arc and Xe