84% on 20GB

Chaperone-Thinking-LQ-1.0 has been released as open source. It is based on DeepSeek-R1-Distill-Qwen-32B, and rather than simple quantization, an inference pipeline was applied together with it.

• 4-bit GPTQ quantization reduced the model size from about 60GB to 20GB.
• GPTQ-based QAT and calibration were used to minimize quantization loss.
• QLoRA fine-tuning on medical/scientific corpora was additionally applied.
• For transparency, it was stated that the adaptive identity layer was removed.

The following benchmark results were presented:

• MATH-500: 91.9
• MMLU: 85.9
• AIME 2024: 66.7
• GPQA Diamond: 56.7
• MedQA: 84%

In particular, MedQA 84% was highlighted, explained as a 4-point gap from GPT-4o at about 88%. It was also claimed that this model runs on a single L40/L40s GPU, and compared to the base DeepSeek-R1-32B, it showed 36.86 tok/s vs 22.84 tok/s, i.e. about 1.6x throughput and about 43% lower median latency.

The goal is to secure an inference model usable in on-premise healthcare environments where data must not leave the premises. The model and related materials were released on Hugging Face, under the CC-BY-4.0 license.