AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
Pinterest reduced embedding search costs by up to 30% by introducing quantization and SSD serving to the Manas platform.
GGUF model uploader Bartowski released research on performance improvements for quantized models using new tensor layout maps.
The INT4 quantized version of the NVIDIA Cosmos3 64B model has been released for CUDA and MLX, enabling local execution.
NVIDIA has released Model Optimizer, which integrates six optimization techniques including quantization and pruning to support checkpoint generation for inference engines such as vLLM and TensorRT-LLM.
Introduces techniques for managing the latency and throughput tradeoffs in LLM inference and expanding the efficient frontier through quantization, speculative decoding, and more.
NVIDIA has released the NVFP4 quantized version of DeepSeek-V4-Pro-0813, supporting a MoE architecture that activates 400B parameters out of a total of 1.65T.
The Qwen3-Coder-Next 4bit model, quantized using the calibration-free TQ method, has been released, reducing disk usage to 41.7GB.
IST-DASLab released a Qwen3.8-27B GGUF model with significantly improved performance over previous versions by applying GSQ and RCO technologies.
Numerous cases have been discovered where GGUF filenames differ from the actual saved bit counts due to quantization logic issues in llama.cpp.
Ollama's KV cache quantization settings are applied globally, which can cause performance degradation for specific models.
ik_llama.cpp has added support for DSpark models and Vulkan quantization features.
A checkpoint quantizing Qwen3.8-27B to NVFP4 using the QUASAR QAT algorithm has been released, maintaining BF16-level performance.
The QAH technique is released, recovering performance higher than the original model after structural compression and 4-bit quantization.
Meta has released MobileMoE, a MoE model with 0.9B active parameters optimized for mobile devices.
The addition of ConvRot quantization to llama-cpp-turboquant brings the quality of Q6 quantized models close to Q8 levels.
Escha Labs released a 2-bit quantized model of Qwen3.8-27B, demonstrating performance equivalent to FP8.
Unsloth released the Qwen3.8-27B Dynamic 3.0 quantized model, improving accuracy.
Liquid AI has released the LFM2.5 Q4_0 models, which minimize quality degradation through Quantization-Aware Distillation (QAD).
An abliterated version of Qwen3.8-27B with censorship removed has been released in MLX format for Apple Silicon.
The DFlash developer released DFlash 2, which supports Qwen 3.8 27B and Muse Glimmer, and submitted a llama.cpp pull request.
Released an MLX quantized version of Qwen3.8-27B with safety alignment removed for Apple Silicon.
In a 24GB GPU environment, Qwen3.8-27B demonstrated the best performance in coding and reasoning tasks.
WarpQuant, a 3.62 bpw quantization method combining Hadamard rotation and Output-Fisher recovery to enhance Qwen 27B performance, has been released.
bitsandbytes developer Tim Dettmers has previewed a new quantization methodology, demonstrating high inference performance.