Can a model be well calibrated but unhelpful?

Yes. Predicting the same correct overall event rate for everyone may provide little discrimination.

Does recalibration require rebuilding the whole model?

Not always. A separate mapping can adjust scores, but its suitability and validation still matter.

Is a 90% confidence statement from a chatbot calibrated?

Not merely because it uses a percentage. The claimed event and comparable outcomes need to be defined and evaluated.

AI Glossary / Governance, ethics and safety / Intermediate

Calibration

Definition

Calibration describes how well predicted values agree with observed outcomes in a defined population and setting. For probability predictions, a calibrated forecast of an event should correspond to the event occurring at approximately that frequency across comparable predictions.

What it means

If a service assigns a 20% probability of a parcel arriving late to many comparable deliveries, about one fifth of those deliveries should be late when the probabilities are well calibrated for that setting. This does not tell us which individual parcel will be late.

Calibration differs from discrimination or ranking ability. A model may correctly rank riskier cases above less risky ones while assigning probabilities that are too high. Another model can reproduce the overall event rate while offering no useful distinction between individual cases.

The word also has other meanings, including calibrating measuring instruments. This entry focuses on predictive calibration, following the statistical sources in the glossary. It covers both assessing calibration and adjusting predictions to improve it.

How it works

Evaluate predictions against outcomes on data appropriate to the intended use. For probability models, a calibration plot compares predicted probabilities with observed event frequencies, often using bins or a smooth curve. A well-calibrated relationship lies near the diagonal, subject to sampling uncertainty.

Check more than the overall average. Overprediction in one range can cancel underprediction in another, and good population-level calibration can conceal differences across relevant subgroups. Small bins can produce noisy estimates; broad bins can hide local problems. State how the curve or summary was calculated.

Calibration methods can fit a mapping from model scores to probabilities, using suitable held-out data or cross-validation. The mapping must not be evaluated only on the same observations used to fit it. Changes in the population or outcome process can invalidate earlier calibration, so the result should be tied to a setting and period.

Three practical examples

A delivery planner checks whether predicted delay probabilities match observed delays before using them to estimate expected disruption.
A support team finds that high-priority probability scores rank cases reasonably but overstate the likelihood that specialist assistance will be needed.
A forecasting analyst examines whether a model's low and high predicted values systematically fall below or above observed averages.

Think of it this way

Think of a weather forecaster whose “one in five” prediction actually occurs about one time in five across many comparable forecasts. The statement has a dependable numerical meaning even though any one day may go either way.

The analogy does not imply that all forecasts labeled 20% are interchangeable under every condition. Calibration can differ by season, location or other relevant context, and a small number of outcomes cannot establish an exact frequency.

Worked example

A correct average can hide wrong probabilities

Suppose an invented test has two groups of 100 delivery predictions. The model assigns 10% delay probability to every case in the first group and 50% to every case in the second. It therefore predicts 10 + 50 = 60 delayed deliveries overall.

In reality, the first group has 20 delays and the second has 40, also totaling 60. The overall average prediction is 30%, matching the observed overall delay rate of 30%. Yet the low-probability group is underpredicted: 10% versus 20%. The high-probability group is overpredicted: 50% versus 40%.

This model is correct in the aggregate for the sample but not across the two prediction levels. A report that checks only the average misses the pattern. The figures are a teaching example; real assessment also needs uncertainty estimates and evidence that the cases represent future use.

Illustrative case study

Fictional Larch Logistics uses delay scores to decide where to assign backup capacity. The model ranks routes well, but a recent operational change makes its stated probabilities too pessimistic. Planners reserve more backup than the observed need suggests.

The team examines calibration on recent representative data, distinguishes ranking quality from probability accuracy and fits a candidate recalibration using a separate development procedure. It then evaluates the adjusted probabilities on untouched later observations.

It also checks whether the change affects different route types similarly. The team does not describe the new mapping as permanently correct. The case illustrates that calibration is tied to the operating population and that useful rankings do not automatically supply reliable numerical probabilities.

Common misconceptions and mistakes

High classification accuracy proves calibration. Correct class labels do not establish that probability values are meaningful.
A model calibrated overall is calibrated for every subgroup. Aggregate agreement can conceal local differences.
Calibration makes uncertain events certain. It describes probabilistic agreement across cases.
A confidence score is automatically a probability. Its interpretation needs definition and evaluation.
A calibration adjustment remains valid forever. Distribution or workflow changes can alter the relationship.

Business use

Calibration matters when numerical predictions drive expected costs, resource allocation or thresholds. Ask whether scores are intended as probabilities and whether their interpretation has been tested on relevant data. A model can be useful for ranking without supporting precise probability-based decisions.

Report the assessment population, period, sample sizes and method. Monitor calibration after material changes and distinguish adjustment data from evaluation data. Also retain other measures: a constant forecast equal to the overall event rate may be calibrated in a limited sense yet offer little help in deciding which individual cases need attention.

Check your understanding

A model predicts 60 events across 200 cases and exactly 60 occur. Does that alone prove its probabilities are calibrated at every level?

Answer: No. Overprediction in some ranges can cancel underprediction in others. Examine the relationship across prediction levels and relevant contexts.

Related terms

Calibration curves
Discrimination
Brier score
Probability prediction
calibration curve
reliability diagram
recalibration
confidence score
distribution shift

Learn next

Read calibration curve and discrimination. Group a set of held-out probability predictions into interpretable ranges and compare each range's average prediction with its observed event rate, retaining sample counts and uncertainty.

Discrimination

Sources

Harrell, Glossary of Statistical Terms, calibration: https://hbiostat.org/glossary/. Scikit-learn, Probability calibration: https://scikit-learn.org/stable/modules/calibration.html. These support the predictive definition and assessment methods. Delivery counts and the logistics case are original illustrations, not reported operational results.

Previous: Calibrated probability assessment Next: Calibration curves