Nota compresses 30B LLM to 10GB with 1.58-bit ternary quantization to run on MacBook

• Nota applied 1.58-bit ternary quantization to NVIDIA Nemotron 3.5 Lightning 30B(A3B), reducing weight size from approximately 60GB to approximately 10GB
• By applying distillation and boundary masking, it recovered 89.4% performance compared to the original on the average of 7 generative benchmarks
• It can run on MacBooks with 24GB or more, and a preview model in MLX format has been released on Hugging Face
• Adopting the PTQ (Post-Training Quantization) approach, it converted existing BF16 models without pre-training, and the identity of the original model was maintained during the recovery process

Technical Approach

• Applied ternary quantization representing weights with only three values: -1, 0, and +1
• Combined on-policy and off-policy distillation to correct output distribution changes caused by quantization error
• Used boundary masking to train only weights near the threshold, reducing noise and stabilizing performance

Performance and Usage

• Recovered to 94% of the original level with scores of 89.31 on GSM8K and 90.85 on HumanEval
• MATH500 and IFEval-ko showed relatively lower recovery rates, but the overall average recorded 83.82 points
• A 16GB MacBook can run it with GPU limit adjustments, but 24GB or more is recommended for system headroom