Dynamic Quantiser Released, Improving Accuracy Over Standard Quantization in VRAM-Constrained Environments

The open-source tool Dynamic Quantiser has been released, performing dynamic quantization on llama.cpp-compatible GGUF files. This tool requires only the GGUF file and llama.cpp, without needing a dataset, and operates by minimizing the cosine deviation of the entire model. Cosine deviation shows a Pearson correlation of 0.99 with KLD (Kullback-Leibler divergence), making it a good indicator of quantization quality.

Key Features and Working Principles

• Precise Dynamic Quantization: Selects the optimal quantization level for individual tensors within a fixed file size to minimize the cosine deviation of the entire model.
• Custom Quantization Support: Finds approximate solutions at the layer/group level, allowing users to manually adjust quantization levels.
• Disk Space Minimization: Also provides a mode that minimizes disk space within a fixed cosine deviation.
• Optimization Algorithm: Optimizes the objective function through the Dinkelbach algorithm, a separable Lagrangian inner loop, and multi-start local search.

Performance and Limitations

• Benchmarks: In tests with the Qwen 3.8 27b model, Unsloth dynamic quantization showed better performance on WikiText, but on a mixed dataset (Bartowski's calibration v5), Dynamic Quantiser occasionally outperformed Unsloth dynamic 3.0 at certain quantization levels.
• Limitations: Overall accuracy is lower than the SOTA dynamic quantization tool Unsloth dynamic 3.0, and IQn_XS style quantization is currently excluded due to inference speed degradation. Initial table building takes approximately 25 minutes on a Ryzen 5 3600, which is a one-time task.
• Development Background: The idea was provided by humans, but the code was written by AI, and it was primarily tested in Windows environments.