AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
WARP has implemented local inference for the 2.78-trillion-parameter Kimi K3 on a 64GB MacBook.
Cactus has released Needle 2, a 14MB on-device agentic LLM.
CKA-QAD improves inference and coding performance by preserving internal representations during NVFP4 quantization-aware distillation.
Unsloth has released the massive GGUF quantized files for Kimi K3.
Quantization, KV cache compression, and inference chips reduce the cost and latency of agentic AI.
LittleBit-2 improves the quantization performance of sub-1-bit LLMs through latent geometric alignment.
Cloudflare reduced GPU memory usage and increased throughput for Kimi and GLM.
MSLK has organized Attention, GEMM, and MoE kernel usage in one place.
NanoQuant compresses a 70B LLM to below 1 bit using a single H100 GPU.
EdgeRazor improves preservation of teacher LLM performance even at 1.88-bit precision.
Escha-W2 was released, quantizing the Qwen3.6-35B-A3B model to 2-bit so it can run even on low-spec GPUs.
BeeLlama.cpp, a fork of llama.cpp, has released v0.4.1, which improves KV cache quantization efficiency.
POCKET-Image achieves accurate Korean text rendering even on-device through a pipeline that preserves glyph shapes.
It proposes SLQ, a statistically lossless quantization technique that increases inference speed while minimizing performance degradation in LLMs.
audio.cpp 0.4 adds Higgs Audio v3 TTS and Fish Audio S2 Pro among others, now supporting 35 model families with Q8 GGUF improving speed and VRAM.
Hugging Face Diffusers has officially integrated Nunchaku W4A4 quantization based on SVDQuant, cutting VRAM usage in half and improving inference speed.
Bad Theory Labs has released BTL-3, a 27B open-weight agentic coding model that runs from a single 8.39GB file, under the Apache-2.0 license.
llama.cpp is preparing an update that boosts prompt processing speed by 15% on ROCm environments and resolves certain quantization bugs.
A study has found that quantizing Qwen2.5-Coder to 4-bit results in almost no degradation in code generation performance.
BeeLlama v0.4.0, a fork of llama.cpp with improved KV cache quantization performance, has been released.
Even with low reconstruction error, downstream task performance can drop, and the optimal compression code varies by consumer.
Through CUDA kernel optimization in llama.cpp, the inference speed of the Bonsai model was improved by about 5%.
PrismML has released Bonsai 27B, a compressed version of the Qwen 27B model using 1-bit and ternary methods that can run even on smartphones.
An inference engine written in C has been released that keeps a large MoE model's weights on disk and streams only the parts needed, allowing it to run even on low-spec PCs.