AI BriefingCohere's CALIBER method cuts calibration error by 52.5% in reasoning models
·2026.10.20 22:40
Key point
The method trains models to report both pre-reasoning and post-reasoning confidence, improving accuracy on math and QA benchmarks.
Details
- CALIBER trains reasoning models to estimate success probability before reasoning and answer correctness after reasoning, reducing calibration error by 52.5% on BigMathDigits for the 7B model compared to the best single-confidence baseline.
- The 7B model also achieves the best AUROC (0.806) and Brier score (0.130) on BigMathDigits, with calibration error reductions of 26.8% on GPQA, 32.2% on TriviaQA, and 36.2% on SimpleQA without additional task-specific training.
- Training uses separate targets: initial confidence is based on the frequency of successful attempts across multiple trials (e.g., 70% if 7 of 10 succeed), while final confidence targets the correctness of the specific generated answer.
- On the 30B model, CALIBER shows a large advantage in AUROC (0.797) over CoCA-lite (0.508) on BigMathDigits, despite similar ECE scores, highlighting the importance of distinguishing correct from incorrect answers.
- Confidence changes during reasoning are context-dependent; mean confidence rises for correct answers (78.5% to 83.5%) and falls for incorrect ones (62.6% to 54.5%) on BigMathDigits, but the final estimate is the primary indicator of answer reliability.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Perplexity SPACE security evaluation finds no sandbox escapes; specialized security models underperform general models