Can people improve calibration?

Training and feedback can help, although improvement varies.

Does calibration eliminate uncertainty?

No. It aims to describe uncertainty more accurately.

Are verbal labels enough?

Labels such as likely need an agreed interpretation before numerical calibration can be assessed.

AI Glossary / Statistics and probability / Advanced

Calibrated probability assessment

Definition

Calibrated probability assessment is the practice of expressing uncertainty with probabilities or ranges whose stated confidence agrees with observed outcomes over an appropriate collection of judgments. Events assigned a probability near 70 percent should occur about 70 percent of the time, allowing for sampling variation and the circumstances being assessed.

What it means

Calibration connects the strength of a claim to evidence about how such claims perform.
For numerical estimates, an assessor might provide a range intended to contain the unknown value 90 percent of the time. Across suitable repeated assessments, coverage should be near 90 percent. These subjective uncertainty ranges should not automatically be interpreted as frequentist confidence intervals for a model parameter; their construction and intended meaning must be explained.

How it works

Define questions with clear resolution criteria and record the probability or range before the answer is known. Retain the date and information available at the time. After resolution, compare stated probabilities with observed frequencies and examine range coverage.
Training can include feedback, reference classes, consideration of opposing evidence, and practice updating judgments. Research on forecasting tournaments has found benefits from probabilistic-reasoning training, but such findings do not guarantee identical improvements for every person or setting.
Assessment should consider both calibration and informativeness. A forecaster who always repeats an overall base rate may be calibrated in aggregate while failing to distinguish relatively likely from relatively unlikely events.

Three practical examples

A project team records probabilities of meeting specific milestones by defined dates. Later reviews compare forecasts with outcomes rather than rewriting confidence after the result is known.
A specialist estimates uncertain quantities using ranges and receives feedback on whether resolved values fell inside them. The review examines both misses and excessively broad ranges.
An AI evaluation team compares model confidence with independently checked correctness. A confident writing style is not itself a validated probability estimate, so the meaning of the reported score must be established first.

Think of it this way

A speedometer is useful when its reading corresponds to actual speed. Similarly, a confidence estimate is useful when its stated level corresponds to how often comparable judgments succeed. A display that always looks decisive is less helpful than one that accurately reflects changing uncertainty.

Worked example

Consider sixty invented binary forecasts, with one forecast per resolved question. Twenty events receive probability 0.2, twenty receive 0.5, and twenty receive 0.8. Suppose the numbers that occur are four, ten, and sixteen, respectively.
The observed frequencies are 4/20 = 0.2, 10/20 = 0.5, and 16/20 = 0.8. Thus the three forecast groups agree exactly with their observed frequencies in this constructed sample. This does not prove permanent calibration: each group has only twenty observations, and another sample may differ.
There are thirty occurring events overall, so the overall event rate is 0.5. A second forecaster who always predicted 0.5 would match that overall frequency but would make no distinctions between the three groups.
Using the binary Brier score, the average squared difference between forecast probability and outcome, the first group's score is [4 times (0.2 minus 1) squared plus 16 times 0.2 squared]/20 = 0.16. The middle group's score is 0.25. The final group's score is 0.16. Averaging equally sized groups gives 0.19.
The constant-0.5 forecaster scores 0.25 on every question, hence 0.25 overall. Lower is better under this scoring convention. The difference demonstrates that matching the overall event rate alone does not capture all useful forecasting information. The Brier score assesses more than calibration alone.
Now split the twenty forecasts at 0.8 into two categories of ten. Suppose all ten events occur in the first category, but only six occur in the second. The pooled rate remains 16/20 = 0.8, while the category rates are 1.0 and 0.6. Aggregate agreement can therefore conceal subgroup differences. These small category counts also warrant caution rather than instant conclusions about stable subgroup behavior.
For a separate range exercise, imagine twenty intervals labeled 90 percent, of which fourteen contain the resolved quantity. Observed coverage is 70 percent. Eighteen covered values would match the nominal rate exactly in this sample, but a calibrated process need not hit exactly eighteen on every set of twenty. Repeated shortfalls would motivate reviewing range construction and the questions being assessed.

Illustrative case study

Fictional company Alder Planning introduces a forecast journal for launch milestones. Each entry names the event, deadline, probability, evidence, and resolution rule. Review meetings retain earlier forecasts when participants update their assessments, making the information timeline visible.
The team checks calibration by probability band and also compares performance across different project types. It avoids treating many updates about one milestone as independent resolved events. Feedback focuses on improving future estimates rather than rewarding unwarranted certainty. This fictional scenario illustrates a review process, not a claim that keeping a journal guarantees calibrated judgment.

Common misconceptions and mistakes

A 90-percent forecast can fail without being a bad forecast. Calibration concerns a collection of judgments, not certainty about one outcome.
Perfect observed agreement in a small sample does not establish a durable ability. Sampling variability and changing conditions matter.
Being calibrated does not mean being maximally informative. Forecasts should also discriminate among events when evidence permits.
Very wide ranges can achieve high coverage while providing little decision value. Coverage must be interpreted alongside width and the stated target.
Checking only resolved or convenient questions can distort the evaluation. Keep an explicit policy for unresolved, canceled, and ambiguous questions.

Business use

Use calibrated assessments when decisions depend on uncertainty rather than a single confident answer. Establish clear events, preserve forecast histories, and provide outcome feedback. Evaluate performance on future questions as well as training exercises. Connect probabilities to the actual decision and its consequences, while keeping forecast quality separate from whether a particular outcome was desirable.

Check your understanding

Why is the 0.8 group calibrated in the example?

Answer: Sixteen of its twenty events occur.

Why is the constant-0.5 forecast less informative?

Answer: It does not distinguish the groups.

Can pooled calibration hide differences?

Answer: Yes; the two categories have observed rates of 1.0 and 0.6 despite pooling to 0.8.

Related terms

Calibration
Probabilistic forecasting
Brier score
Resolution
overconfidence
reference-class forecasting
uncertainty interval
reliability diagram

Learn next

Study proper scoring rules, calibration plots with uncertainty, and the distinction between calibration and resolution. Practice recording forecasts before outcomes are available.

Sources

Chang, Chen, Mellers, and Tetlock, Developing expert political judgment, 2016: https://dlab.sauder.ubc.ca/sjdm/journal/16/16511/jdm16511.html . Hubbard Decision Research, The Principles of Applied Information Economics, uncertainty-assessment training material: https://hubbardresearch.com/wp-content/uploads/2020/11/HTMA-The-Principles-of-AIE-Module-3-PDF-20-Nov-2020.pdf . Forecast counts, calculations, and company scenario are original illustrations.

Previous: Calculus of variations Next: Calibration