AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
Shard has been released, compressing the KV cache memory of Llama-3.1-8B by about 10x without performance degradation.
A CUDA-based FWHT has been added to llama.cpp, improving inference speed when using KV-cache quantization.
A 2-bit quantization technique has been released that compresses KV cache memory usage by about 7x while minimizing performance degradation.
The SDK 'Cider,' which supports W8A8 activation quantization on Apple Silicon by addressing MLX's limitations, has been released.
Huawei unveiled the BitCPM-CANN model series, which supports 1.58-bit quantization-aware training on Ascend NPUs.
llama.cpp now supports NVIDIA FP4 quantization and Multi-Token Prediction (MTP) features.
A problem where CUDA acceleration is limited when using asymmetric KV cache quantization in llama.cpp, along with solutions, is being discussed.
Cohere has released Command A+, an Apache 2.0-licensed model featuring lossless quantization and native citation capabilities.
A study on Mix-Quant has been published, improving efficiency by applying aggressive quantization to the prefill stage while keeping the decoding stage at high precision
Tencent has unveiled the Hy-MT2 series of multilingual translation models supporting 33 languages.
Cohere has released Command A+, an open-weight model based on Apache 2.0.
ByteShape released Qwen 3.6 35B GGUF NTP and MTP quantizations along with benchmarks.
Cohere has open-sourced Command A+, a 218B-parameter MoE model, under the Apache 2.0 license.
Maximum activation values across 27 open LLM checkpoints varied significantly depending on architecture and training stage.
OpenBMB released BitCPM4-CANN 1B, 3B, and 8B on Hugging Face.
DwarfStar 4, a Metal-based local inference engine, has been released supporting DeepSeek V4 Flash.
vLLM concluded that FP8 KV-cache is the default choice, while TurboQuant's higher-bit options offer little real benefit.
cyankiwi has released an AWQ 4-bit update that jointly optimizes scales and quantization ranges to reduce error.
NVIDIA released NVFP4 versions of Kimi-K2.6 and Kimi-K2.5.
PyTorch 2.12 significantly strengthens CUDA performance, export, and ROCm support.
OpenAI summarized lessons on research challenges in the age of AI agents through Parameter Golf.
The number of GGUF-format model uploads on Hugging Face has roughly doubled over the past two months.
ExLlamaV3 has added Gemma 4 support, DFlash optimization, and bug fixes.
The official DeepSeek V4 paper has been released, adding FP4 QAT and stabilization techniques.