AI Glossary / Causal inference and experimentation / Intermediate
Brier score
Definition
The Brier score measures the quality of probability predictions by averaging squared differences between predicted probabilities and observed outcomes. For a binary event, the common version uses outcomes coded as 0 or 1 and computes the mean of (probability minus outcome) squared. Lower is better. It evaluates probability forecasts rather than just their ranking or the class labels obtained after applying a threshold. Binary and multiclass scaling conventions must be stated when reporting scores.
What it means
A prediction of 0.8 says more than “yes”: it expresses an estimated event probability. The Brier score gives that forecast a loss of (0.8 - 1) squared, or 0.04, if the event occurs, and (0.8 - 0) squared, or 0.64, if it does not. A confidently wrong forecast therefore receives a larger penalty than a less confident wrong forecast. Evaluation across many appropriate cases matters because one realized outcome cannot by itself establish whether a probability was reasonable.
How it works
Collect held-out probability forecasts and their eventual outcomes, ensuring the probability refers to the same event and time horizon as the label. Calculate the squared error for each case, then average using declared weights if needed. For multiple mutually exclusive classes, a common version sums squared errors across the probability vector and its one-hot outcome vector before averaging cases. That unscaled multiclass score ranges from 0 to 2; the familiar binary one-probability form ranges from 0 to 1. Keep label order and scaling explicit.
Three practical examples
- Delivery planning: evaluate predicted probabilities of arrival by a specified deadline after outcomes become available.
- Support operations: compare probability forecasts that a ticket will need specialist review, using a consistent outcome definition.
- Model recalibration: assess held-out probability quality before and after a calibration procedure, alongside reliability plots and ranking measures.
Think of it this way
Think of a forecaster placing a marker between “will not happen” and “will happen”, then measuring squared distance to the outcome once it is known. Repeating this across cases evaluates the forecasts. The analogy has a limit: a good probability forecast need not sit at the eventual outcome every time, because uncertainty can be real rather than a model defect.
Worked example
Suppose four binary outcomes are 1, 0, 1, and 0. A model assigns positive-event probabilities 0.8, 0.3, 0.6, and 0.1. Its squared errors are 0.04, 0.09, 0.16, and 0.01, summing to 0.30. The binary Brier score is 0.30/4 = 0.075. A constant forecast of 0.5 produces squared error 0.25 for every case, so its score is 0.25 on this sample.
At a threshold of 0.5, the first model classifies all four outcomes correctly. Yet its Brier score is not zero because its probabilities are not all exactly 0 or 1 matching the outcomes. That is not a reason to force extreme probabilities: future uncertainty may make such certainty harmful. If a forecast of 0.99 is followed by outcome 0, its squared error is 0.9801. The example demonstrates the distinction between label correctness and probability loss, but four cases are far too few to establish robust superiority or calibration in a real population.
Illustrative case study
A fictional service-planning team compares two probability models with similar classification accuracy. One gives very extreme predictions and occasionally makes confident errors. The other uses less extreme values and obtains a lower held-out Brier score. The team also examines reliability by probability range and by ticket type, because the overall score combines more than calibration alone. It checks that delayed outcomes have been observed consistently before comparing versions. This turns a probability dashboard into a documented assessment of forecasts with a clear event and horizon.
Common misconceptions and mistakes
A lower Brier score does not automatically mean better calibration alone. The score also reflects the ability to distinguish cases with different event probabilities and the uncertainty of the evaluated outcomes. Scores from populations with different event frequencies are not directly comparable without context. Zero loss on a small sample is not proof of perfect future forecasting. The Brier score is also not a business cost function: the same squared error may lead to very different consequences depending on the decision made from the forecast.
Business use
Use the score when probabilities themselves inform planning, prioritization, or thresholds. Compare candidates on the same held-out cases and include a meaningful baseline forecast. Report the outcome definition, time horizon, event frequency, weighting, and scaling. Inspect calibration curves and error slices rather than relying on a single average. If probabilities are recalibrated, fit that mapping on suitable separate data and preserve an independent assessment. Connect probability quality to the intended decisions through a separate analysis of capacity and error consequences.
People also ask
Why is it called a proper scoring rule?
In expectation, the binary score is minimized by reporting the true event probability rather than deliberately distorting it.
Can it be used with hard labels?
Yes mathematically, but replacing probabilities with 0 or 1 discards information and turns binary squared error into ordinary misclassification error.
Is there one universal good Brier score?
No. Interpretation depends on the population, baseline, task, sample size, and scale convention.
Check your understanding
Two models make identical class predictions at threshold 0.5. Must their Brier scores match?
Answer: No. Their probabilities can differ substantially while staying on the same side of the threshold. Brier scoring evaluates those probability values, not just the resulting labels.
Related terms
- Calibration
- Log Loss
- Discrimination
- Probability forecast
- reliability diagram
- proper scoring rule
- event prevalence
- squared error
- Brier skill score
- decision threshold
Learn next
Calculate the score for several probability forecasts against the same binary outcomes. Compare a constant baseline, a cautious forecast, and an overconfident forecast. Then study reliability diagrams and the calibration-resolution-uncertainty decomposition to understand why an improved overall score does not isolate which aspect of probability forecasting improved.
More in Causal inference and experimentation
- Balanced accuracy score - Balanced accuracy score measures classification performance by averaging the recall of each class with equal class weight.
- BLEU (Bilingual Evaluation Understudy) - BLEU, short for Bilingual Evaluation Understudy, is an automatic metric originally designed to compare machine translations with human reference translations.
- Box-Benkhen experiment - A Box - Behnken experiment uses a response-surface design to study how several controllable factors affect an outcome and to fit a quadratic model.
- Calinski-Harabasz Index - The Calinski-Harabasz Index is an internal clustering score that compares separation between cluster centers with variation inside clusters, adjusted for...
- Causal estimand - A causal estimand is the precisely defined causal quantity that an analysis aims to learn.
- Causal graphs - Causal graphs are diagrams that represent assumptions about causal relationships among variables.
Work in finance? Get the free CFO AI Prompt Pack: 7 prompts for finance leaders.
Sources
Scikit-learn, brier_score_loss: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html Scikit-learn, probability calibration and Brier decomposition discussion: https://scikit-learn.org/stable/modules/calibration.html Scikit-learn, model evaluation guide: https://scikit-learn.org/stable/modules/model_evaluation.html
Previous: Bridging inference Next: Brill tagger