OpenBonsai open-sources pipeline to compress BF16 models to ~2 GB ternary GGUFs

·2026.10.29 17:13

Key point

Apache-2.0 framework enables post-training ternary quantization of custom fine-tunes, reducing a 9B model to under 2 GB.

OpenBonsai is an Apache-2.0 framework that compresses BF16 models into Prism's PTQ1_0 GGUF format using ternary weights (-1, 0, +1).
Compresses Qwen3.5-9B from ~18 GB to 1.997 GB in approximately 5 hours on a single RTX 5090 (32 GB).
Pipeline uses Signed Hadamard rotations, Sequential GPTQ, FP16 group scales, and self-generated reasoning traces for calibration.
Current compressed model scores 65/100 on GSM8K (thinking off) and 34/50 on HumanEval (thinking off), lagging behind Prism's official models.
Research preview status: requires 32 GB GPU for compression and Prism's llama.cpp fork for inference; upstream llama.cpp and WebGPU are not yet supported.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.