AI Glossary / AI and machine learning / Advanced
Calibration curves
Definition
A calibration curve checks whether a model's confidence means what it says: when the model predicts a 70 percent chance for a group of cases, about 70 percent of those cases should actually happen.
What it means
Many models output probabilities: a 0.8 chance this lead converts, a 0.1 chance this loan defaults. Those numbers are only useful for planning and pricing if they are honest, and honesty is exactly what calibration measures.
Accuracy and calibration are different questions. A model can rank customers well, putting likely converters above unlikely ones, while its stated probabilities are systematically too high or too low.
A well-calibrated model's probabilities match observed frequencies. Overconfident models predict higher chances than reality delivers; underconfident ones do the reverse. Both quietly distort any decision that uses the number.
Calibration matters most where the probability itself is the product: risk pricing, churn prioritization, medical triage and any place a person reads the score as a likelihood.
For a business, a calibration curve is the difference between a score you can plan with and a score that merely sorts. Sorting helps you call the best leads first; calibration tells you how many will actually buy.
The picture is often called a reliability diagram, and the numbers behind it are summarized as expected calibration error, a single score averaging the gap between predicted and observed across the bands.
How it works
Gather predictions on cases where the true outcome is now known, and sort them into bands by predicted probability, such as 0.0 to 0.1, 0.1 to 0.2 and so on.
For each band, compare the average predicted probability with the actual frequency of the outcome. A band predicted at 0.3 should contain about 30 percent real cases.
Plot predicted against actual. A perfectly calibrated model lies on the diagonal; above or below it shows under- or overconfidence, and where in the range the problem lives.
If needed, recalibrate: fit a small adjustment, such as Platt scaling or isotonic regression, that maps raw scores onto observed frequencies using held-out data.
Recheck after retraining, data drift or any model change, because calibration decays as the world moves away from the training data.
Three practical examples
- Lead scoring: leads scored 0.8 should convert about eight times in ten, or the sales plan built on them is fantasy.
- Credit risk: priced default probabilities must match realized default rates, or the pricing loses money in exactly one direction.
- Demand planning: a 30 percent stockout risk should mean stockouts in about three of ten comparable weeks.
Think of it this way
Think of a weather forecaster who says 80 percent chance of rain every day for a month. If it rained on only half of those days, the forecasts were confident and wrong. Calibration is the audit of that claim.
The limit: calibration says nothing about whether the 80 percent days were the right days. A model can be calibrated and still a poor discriminator; you need both checks.
Worked example
recalibrating a lead scorer before the sales plan
Suppose a marketing team scores leads from 0 to 1. Leads scored around 0.8 actually convert at 0.5, while leads around 0.2 convert at 0.3. The ranking is fine; the numbers are inflated at the top.
The team plots the calibration curve, sees the bow above the diagonal, and fits a recalibration layer on held-out data. Figures are illustrative.
After recalibration, a 0.6 lead really converts about six times in ten, and the revenue plan is rebuilt on the corrected numbers.
The ranking of leads barely changed, but the forecast did, and the forecast is what finance reads.
Illustrative case study
A fictional lender prices loans from a model's default probabilities. A calibration review finds the model overconfident among the riskiest applicants, predicting 15 percent default where 25 percent occurs.
Recalibration raises prices for that segment and quietly fixes a slow leak in the loan book. The board now asks for the calibration curve alongside every accuracy figure.
The invented lesson: when money is priced off a probability, uncalibrated confidence is a cost that hides inside average performance.
Common misconceptions and mistakes
- A calibrated model is an accurate one. Calibration is about honest probabilities; accuracy is about right answers, and one does not give you the other.
- Calibration is all or nothing. A model can be calibrated in one range and overconfident in another, which the curve reveals.
- Recalibration fixes a weak model. It repairs the probabilities, not the model's power to tell cases apart.
Business use
Treat any probability from a model as unproven until you have seen its calibration curve on data like yours.
Ask vendors for calibration evidence, not just accuracy, whenever the probability itself feeds pricing, planning or promises.
Make calibration review part of model sign-off whenever probabilities feed prices, staffing or promises, and repeat it on a schedule, because drift quietly bends the curve.
People also ask
What is perfect calibration?
Across many bands, predicted probabilities match observed frequencies, and the curve sits on the diagonal.
Can calibration be fixed after training?
Often yes, with a recalibration layer fitted on held-out data, without retraining the base model.
Does recalibration change the ranking?
Usually not much; it rescales the scores so the numbers match reality while the ordering stays.
How many cases do I need per band?
Enough for the observed frequency to be stable; thin bands wobble, so hundreds per band is a rough guide.
Check your understanding
A model predicts 90 percent for a group of 100 cases and 60 of them happen. What does the calibration curve show at that point?
Answer: The point sits below the diagonal: predicted 0.9 against observed 0.6, evidence of overconfidence in that range.
Related terms
Learn next
AI for Business Owners explains how to read model scores before you plan or price on them: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]
More in AI and machine learning
- Brown clustering - Brown clustering is an unsupervised method that groups words into a hierarchy using patterns of neighboring words in a text collection.
- Bucketing - Bucketing groups values into ranges, or buckets, such as ages 18 to 24 and 25 to 34, so a model or report works with a few clear groups instead of thousands...
- C4.5 algorithm - C4.5 is a supervised learning algorithm developed by Ross Quinlan for building classification decision trees from labeled examples.
- Candidate generation - Candidate generation is the fast first stage of a recommendation or search system, which picks a few hundred plausible items out of millions so a slower...
- Canonical correspondence analysis - Canonical correspondence analysis is a constrained ordination method that relates a table of species abundances, or similar non-negative response data, to...
- Cascading classifiers - Cascading classifiers are classification stages arranged so that an example's result at one stage determines whether it proceeds to another.
Sources
[S20] scikit-learn documentation, Probability calibration. Full URLs are in the source ledger. Verified 11 October 2026.
Previous: Calibration Next: Calibration frames (bias, dark, flat)