AI Briefing#quantization
The latest AI and developer news about #quantization, with the original source and a short summary.
The latest AI and developer news about #quantization, with the original source and a short summary.
On Gemma 4 26B-A4B GGUF, Unsloth quants top the charts by KLD in most cases.
Unsloth's Mistral Small 4 GGUF has been updated along with chat template fixes.
Compressed the KV cache on Qwen 3.6's 1M context, reducing it from 10.7GB to 6.9GB.
Explains the bit composition, value range, and E2M1 format of FP4.
Qwen 3.6 35B-A3B has been quantized for Ollama, supporting even tools and thinking.
Uniform quantization breaks Qwen3.5's sensitive layers.
Boosted Qwen3.6-35B-A3B speed by 54% on an RTX 5070 Ti using --n-cpu-moe.
In a 30k LOC agentic evaluation, Qwen 3.6 35B outperformed Gemma 4 26B.
On dual RTX 5060 Ti GPUs, Qwen 3.6-35B-A3B hit 21.7 tok/s at 90K context.
SwiftLM presented results of running a 35B Qwen model on a 24GB Mac with 40K~100K context.
In the Qwen3.6-35B-A3B GGUF benchmark, Unsloth quants claim the best size efficiency relative to KLD.
Qwen3.6-35B-A3B-UD-Q2_K_XL handled 58 tool calls at a 98.3% success rate on a 16GB VRAM laptop.
A locally run Qwen3.6-35B-A3B drew a better pelican SVG than Opus 4.7.
Ternary Bonsai has released 8B, 4B, and 1.7B models using 1.58-bit ternary weights.
On an M2 Ultra 64GB, Qwen3.5-35B-A3B Q8_K_XL achieved 63 t/s in generation.
The uncensored Qwen3.6-35B-A3B Aggressive version and K_P GGUF quants have been released.
PrismML has released Ternary Bonsai, a family of 8B, 4B, and 1.7B LLMs using 1.58-bit ternary weights.
A case of running Qwen 3.6 35B on an RTX 2000 Ada 16GB at 400 t/s prompt and 24 t/s generation.
Using turbo3, achieved about 40 tok/s running Qwen3.6 35B A3B at 260k ctx on a 12GB VRAM 3080.
A benchmark running MiniMax M2.7 AWQ on 8x MI50 32GB hit a peak of 64 tok/s.
DFlash doesn't speed up as much as expected with MoE, quantization, concurrent use, and long context.
Compared MMLU scores of GGUF quantized models in a 24GB VRAM environment.