AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
Unsloth has released NVFP4 quantized models for Qwen3.6 that boost inference speed by up to 2.5x on Blackwell GPUs.
llama.cpp's ggml library has added Q2_0 quantization for CPU to support Ternary Bonsai models.
The Sana 1.6B text-to-image model was quantized to 1.58-bit ternary, reducing its size by 8.6x.
EdgeRazor is a lightweight LLM framework based on mixed-precision quantization and distillation that can be deployed across diverse hardware.
OSCAR, a technique using spectral covariance-aware rotation to reduce quantization loss in KV Cache, has been released.
This introduces case studies that used the Gemma 4 model to build an on-device English tutoring app and visual intelligence-based services.
The ggml-webgpu implementation in llama.cpp significantly improved inference speed through matmul optimization for k-quants models.
A new implementation of NanoQuant, which can compress transformer models to sub-1-bit, has been released.
MTP heads for Gemma 4 QAT models have been released, along with news of a fix for a llama.cpp parallel execution bug.
They unveiled **InstinctRazor**, a technology optimized to run MoE models on edge devices.
Google has released Gemma 4 models with QAT technology applied to boost efficiency on mobile and laptop environments.
Unsloth has released **MTP GGUF weights** for the 31B, 26B-A4B, and 12B models of Gemma 4.
A feature has been proposed for llama.cpp that dynamically changes the KV cache quantization level without reloading the model.
DeepLearning.AI has released a course on optimizing efficient LLM inference using vLLM.
Huawei has unveiled KVarN, a new KV-cache quantization backend that improves both capacity and throughput for vLLM at the same time.
**OCTOPUS**, which maximizes transformer KV Cache compression performance by combining octahedral parameterization with joint quantization, has been released.
Bonsai Image 4B, an ultra-lightweight image generation model that reduces memory usage to around 1GB through 1-bit and Ternary quantization, has been released.
The next-generation computer use agent series Holo3.1, delivering strong performance in local and mobile environments, has been released.
llama.cpp has fixed an issue where quantized KV cache did not work when using --sm tensor in multi-GPU environments.
Ultra-lightweight 1-bit and Ternary Bonsai Image 4B models have been released, enabling high-quality image generation on local devices.
A Qwen3.6 GGUF model has been released that integrates an MTP head, enabling self-speculative decoding without a separate draft model.
NVIDIA has released a quantized version of Alibaba's Qwen3.6-35B model using the NVFP4 data type.
A native HIP W4A16 kernel that optimizes performance on AMD GPUs has been added to vLLM.
PrismML has unveiled Bonsai Image 4B, a 1-bit/ternary-based model that can run locally in the browser via WebGPU.