Technique for Calibrating LLMs' Verbal Confidence

LLMs recognize with high accuracy (0.76–0.88 AUROC) whether their answers are correct through their internal states (hidden states), yet tend to show 99% confidence for all questions when actually answering. This is due to a 'text bottleneck phenomenon' where the model internally knows whether the answer is correct but fails to express this in language.

The researchers addressed this problem through Probe-targeted fine-tuning (LoRA). By using the model's internal probe output as the target for fine-tuning, they trained the model to verbally express the confidence it already internally recognizes.

Key findings:

• Generality: Effectiveness was verified across 8 models (4 model families) ranging from 7B to 70B in scale.
• Proof of causality: Through activation patching, a causal relationship was confirmed where replacing hidden states at specific positions changes confidence.
• Discovery of the bottleneck: In the case of the 70B model, it was observed that while valid metacognitive signals exist in the softmax distribution, the model becomes fixed at 99% confidence at the text output (argmax) stage.

The research code is available on GitHub, and the paper is published on Zenodo.