Is a lower ECE always a better model?

No. It indicates better agreement under that metric, but the model may still have poor accuracy, weak discrimination or hidden subgroup errors.

How many bins should be used?

There is no universally correct number. The choice trades resolution against sample noise and should be stated and checked for sensitivity.

Can calibration change without accuracy changing?

Yes. Adjusting probabilities can change confidence while preserving the most likely class, leaving classification accuracy unchanged.

AI Glossary / Probabilistic machine learning / Intermediate

Expected calibration error

Definition

Expected calibration error (ECE) is a summary measure of how closely a classifier's stated confidence matches its observed correctness. A common empirical version groups predictions into confidence bins, calculates the absolute gap between average confidence and accuracy in each bin, and averages those gaps weighted by the number of predictions.

What it means

If a model assigns 80% confidence to many predictions, good confidence calibration means roughly 80% of such predictions should be correct in the relevant population. Accuracy alone does not establish this: two models can make the same class predictions but attach very different confidence values.

ECE measures a particular aspect of probability quality. In the common top-label version, confidence is the probability assigned to the predicted class. Other versions assess classwise probabilities or use different bins and norms. A report must specify the definition rather than assuming every value called ECE is directly comparable.

How it works

Choose a labeled evaluation sample and partition confidence values into bins. For each nonempty bin, compute its fraction of correct predictions and its mean predicted confidence. Take the absolute difference, multiply by the bin's sample fraction, and sum over bins.

In symbols, the usual binned L1 ECE is sum over bins of (bin size/sample size) × |bin accuracy − bin confidence|. Empty bins contribute zero weight. The result is commonly reported between zero and one or as percentage points.

This is a finite-sample, binned estimate of calibration discrepancy. Bin count, boundaries and sample size affect it. Opposing errors can cancel within one bin before the absolute value is taken. A small aggregate value can therefore hide important local or subgroup problems.

Three practical examples

1. A support-ticket classifier's confidence controls whether a prediction receives manual review. Analysts check whether high-confidence predictions are correct as often as claimed.

2. A product-image model is evaluated before and after a probability-calibration step. ECE is reported alongside accuracy and a reliability diagram, using independent evaluation data.

3. A model-monitoring team recomputes calibration diagnostics after a change in the customer mix. A previously low ECE does not establish calibration under the new distribution.

Think of it this way

Think of a weather forecaster's confidence ledger: among many statements made with similar confidence, how often did the stated event occur? ECE compresses discrepancies across confidence groups into one number. The analogy is limited because top-label classification ECE concerns whether the chosen label is correct; a weather-event probability may instead call for event-specific calibration, and a single score cannot show every pattern in the ledger.

Worked example

Consider ten predictions divided into two bins. The lower-confidence bin contains four predictions, each with confidence 0.60; three are correct. Its accuracy is 3/4 = 0.75 and its absolute gap is |0.75 − 0.60| = 0.15.

The higher-confidence bin contains six predictions, each with confidence 0.90; three are correct. Its accuracy is 3/6 = 0.50 and its gap is |0.50 − 0.90| = 0.40.

ECE is (4/10) × 0.15 + (6/10) × 0.40 = 0.06 + 0.24 = 0.30, or 30 percentage points. Overall accuracy is 6/10 = 0.60. Overall average confidence is (4 × 0.60 + 6 × 0.90)/10 = 0.78.

Their overall absolute difference is only 0.18, not 0.30, because underconfidence in the first bin partly cancels overconfidence in the second when pooled. This illustrates why binning matters. Ten observations are far too few to treat the calculated value as a precise population calibration measure.

Illustrative case study

Fictional case: Queue Harbour uses confidence thresholds to route documents for review. Its classifier is accurate enough on average, but a reliability diagram shows that predictions near 90% confidence are less reliable than staff expect. Analysts fit a calibration transformation on a dedicated validation set and evaluate it on a separate held-out set.

They also inspect rare document categories and the high-confidence region that drives automatic routing. The team keeps those detailed checks even if overall ECE decreases. This fictional scenario describes an evaluation process, not a measured calibration improvement or an endorsement of a particular threshold.

Common misconceptions and mistakes

ECE is not the fraction of incorrect predictions. A 0.05 ECE does not mean 95% accuracy, nor does it guarantee that every confidence value is accurate within five percentage points.

Zero binned ECE on a finite dataset does not establish perfect population calibration. Broad bins can hide opposing discrepancies, while narrow bins may be noisy. Class imbalance can conceal errors for rare groups because the weighting follows sample frequency.

Top-label ECE also ignores parts of the full probability vector. Reporting it alone can miss poor probabilities for nonselected classes or different kinds of decision-relevant uncertainty.

Business use

Use ECE as one diagnostic when confidence influences operational choices. Specify the binning rule, whether top-label or classwise probabilities are assessed, the norm and the evaluation population. Preserve the same conventions when comparing versions.

Pair it with reliability plots, counts per bin, accuracy and suitable probabilistic scores such as log loss or Brier score. Inspect the regions and groups that matter for actual decisions. Avoid fitting a calibrator and reporting final ECE on the same observations. Reassess after distribution changes, because calibration is a relationship between predictions and outcomes in a particular population.

Check your understanding

If the worked example is placed into one bin, what binned ECE results?

Answer: |0.60 − 0.78| = 0.18. The smaller value arises from the changed binning and cancellation, not from any improvement in the predictions.

Related terms

Probability calibration
Temperature Scaling
Brier score
Log Loss
reliability diagram
confidence
classwise calibration
maximum calibration error

Learn next

Build a reliability diagram with sample counts and compare several binning choices. Then examine proper scoring rules and subgroup calibration to understand what a single ECE value cannot establish.

Sources

Primary sources: Guo and colleagues, On Calibration of Modern Neural Networks, definitions of confidence, bins and ECE: https://proceedings.mlr.press/v70/guo17a/guo17a.pdf ; Nixon and colleagues, Measuring Calibration in Deep Learning, primary abstract: https://arxiv.org/abs/1904.01685 . Independent review checked Guo’s bin definitions and ECE equation (3). The ten-prediction calculation and fictional workflow are original illustrations.

Previous: Expectation propagation Next: Expected frequencies