Know When to Abstain

A toy model answers 1,000 questions with stated confidences. Read its reliability diagram, fix overconfidence with temperature scaling, and choose when it should say 'I don't know' under different penalties for wrong answers.

From Chapter 16: Evaluation, Reliability, Safety & Interpretability