Calibration

What confidence should mean

Suppose a model attaches a confidence to each answer. Collect every answer it gave with confidence near 0.8. If about 80% of them are right, and the same holds at every level, the model is calibrated. Calibration says nothing about how often the model is right; it says whether its confidence is honest.

Measuring it

A reliability diagram sorts answers into bins by confidence (0–0.1, 0.1–0.2, …) and plots each bin's accuracy against its average confidence. A calibrated model's bars sit on the diagonal; bars below it mean overconfidence. Expected calibration error summarises the gaps:

$$\mathrm{ECE} = \sum_b \frac{n_b}{n}\,\big|\,\mathrm{acc}(b) - \mathrm{conf}(b)\,\big| .$$

ECE depends on the binning and on sample size: even a perfectly calibrated model shows a few points of ECE on a thousand answers, from noise alone.

Overconfidence and its cheap fix

Guo and colleagues found that a 1998 LeNet's confidence closely matched its accuracy while a 110-layer ResNet's average confidence was substantially higher than its accuracy, and that temperature scaling, dividing the logits by a single number $T$ fitted on held-out data, was surprisingly effective. Established $$\hat p = \sigma\!\left(\frac{z}{T}\right) \quad\text{(or softmax}(z/T)\text{ for many classes)}.$$

$T > 1$ softens overconfident predictions. Because dividing by a positive number keeps the order of the logits, the predicted class, and so the accuracy, does not change.

Language models

Kadavath and colleagues found larger models well calibrated on multiple-choice and true/false questions in the right format, and that models could estimate the probability that their own proposed answer was true. Established The GPT-4 technical report shows the pretrained model highly calibrated on a subset of MMLU (ECE 0.007) and the post-trained model much less so (ECE 0.074). Established Training toward answers that people prefer rewards sounding sure, which pushes stated confidence toward the extremes. Interpretation On the expert questions of Humanity's Last Exam, all models tested had RMS calibration errors above 70%. Established

For a chat model, "confidence" can come from token probabilities (when available), a number the model states, or agreement among several sampled answers. Each needs its own check.

Tiny example (lab numbers)

The lab gives a toy model 2,000 questions with known true probabilities of being right, fits on 1,000 and reports on the other 1,000. The "pretrained" model states its true probability and has an ECE of 0.031, which is just sampling noise. The "chat-tuned" model ranks questions identically but pushes its confidence toward 0 and 1: ECE 0.114, with 399 of 1,000 answers claiming over 90% confidence. Temperature scaling fits T = 2.17 and brings ECE back to 0.032. Accuracy is 59.2% in all three cases.

Mini experiment

In the lab, choose the chat-tuned model and look at the top bin of the reliability diagram: what fraction of its "over 90% sure" answers are right? Then turn on temperature scaling. Why does one number fix it here, and what kind of miscalibration could a single temperature not fix?