AI BriefingBonsai 27B: On-Device Execution via 1-bit/Ternary Quantization
Key point
PrismML has released Bonsai 27B, a compressed version of the Qwen 27B model using 1-bit and ternary methods that can run even on smartphones.
Details
Bonsai 27B, released by PrismML, is a research output based on the Qwen 2.5-27B model that goes beyond the limitations of existing quantization methods by applying binary (1-bit) and ternary representations.
Existing mixed-precision quantization methods had to leave high-precision tensors in place to prevent quality degradation when going below 4-bit, but Bonsai succeeded in compressing key components such as embeddings, attention, and MLP projections end-to-end to low-bit.
Key Features and Performance:
- Ultra-lightweight: The ternary model has a memory footprint of about 5.9 GB, and the binary model about 3.9 GB, making it runnable on smartphones like the iPhone 17 Pro Max or ordinary laptops.
- Performance preservation: The reasoning, tool calling, and agentic workflow capabilities of the 27B-scale model are largely preserved even in the low-bit environment.
- Practical approach: Unlike methods like BitNet that train in low-bit from scratch, Bonsai adopts an approach that converts already-validated pretrained models to low-bit, enhancing scalability.
- License: All weights are released under the Apache 2.0 license.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.