Is Brier loss lower-is-better?

Under the squared-error convention used here, yes. Some scoring presentations reverse signs or use different scaling, so check the definition.

Does a proper score guarantee calibration?

No finite evaluation guarantees it. Proper scores encourage accurate probabilities in expectation, but calibration still needs assessment and depends on model and data limitations.

Can a model with lower classification accuracy have a better probability score?

Yes. Class labels discard probability detail, so the two measures can rank models differently.

AI Glossary / Causal inference and experimentation / Intermediate

Proper accuracy scoring rule

Definition

A proper accuracy scoring rule evaluates probability forecasts so that reporting the true outcome probabilities achieves the best expected score. Depending on convention, better means a higher reward or a lower loss. A strictly proper rule makes that optimum unique. The term concerns the accuracy of probability assessments, not merely the percentage of class labels predicted correctly. Brier loss and logarithmic loss are familiar examples for categorical outcomes.

What it means

Suppose an event has a 70% chance of occurring given the available information. A probability forecast should be rewarded for saying 70%, rather than encouraged to exaggerate to 100% merely because the event is more likely than not.

A proper scoring rule provides that incentive in expectation over possible outcomes. One realized outcome cannot prove that a probability was correct: an event forecast at 70% can still fail to occur. The rule's defining property concerns expected performance across the outcome distribution.

Proper describes the rule. It does not certify a fitted model, dataset representativeness, or suitability for every business decision.

How it works

A scoring rule takes a predicted probability distribution and the outcome that actually occurs, then assigns a numerical score. For binary outcomes, a Brier-loss convention uses (p − y)^2, where p is the predicted probability of the event and y is 1 if it occurs and 0 otherwise.

To understand propriety, imagine averaging that loss over outcomes generated by the true distribution. The expected loss is minimized by the true probability. Strict propriety means a different probability cannot achieve the same minimum, within the rule's stated domain.

Evaluation averages realized scores over an appropriate sample. Report the sign, normalization, and outcome encoding, because scoring conventions differ. Compare models on held-out data and complement aggregate scores with calibration and subgroup checks. Proper scoring encourages sound probabilities, but finite samples and imperfect model classes still limit what can be learned.

Three practical examples

1. Demand-event forecasting: a team scores predicted probabilities that a daily order count exceeds a capacity threshold, retaining uncertainty instead of evaluating only yes-or-no labels.

2. Subscription forecasting: analysts compare probability models using Brier loss before applying a separate decision policy for customer outreach.

3. Categorical prediction: developers use logarithmic loss to evaluate how much probability a model assigns to the class that actually occurs, rather than only whether it ranked that class first.

Think of it this way

Think of a forecasting contest whose points reward an honest assessment of uncertainty over many rounds. A contestant should not gain an expected advantage by overstating confidence. The analogy is limited because real participants may not know the true probabilities, and a finite contest can reward a lucky forecast. Propriety describes the incentive built into the rule, not certainty about any individual contestant's knowledge.

Worked example

Suppose a fictional binary event has true probability q = 0.70. If a forecast reports p, its expected Brier loss is 0.70 × (1 − p)^2 + 0.30 × p^2.

Reporting p = 0.70 gives 0.70 × 0.09 + 0.30 × 0.49 = 0.063 + 0.147 = 0.21. Reporting p = 1 gives 0 + 0.30 = 0.30. Reporting p = 0.50 gives 0.25. The truthful 0.70 forecast has the smallest expected loss among these candidates.

Indeed, the expression can be rewritten as (p − 0.70)^2 + 0.21, showing that its unique minimum occurs at p = 0.70. However, if the event occurs on one trial, p = 1 receives loss zero while p = 0.70 receives 0.09. That single outcome does not contradict propriety: the guarantee concerns expected loss, including the possibility that the event does not occur.

Illustrative case study

Fictional case: Spruce Services compares two probability models that both classify every score above 0.50 as positive. Their predicted labels are identical, so their ordinary classification accuracy is identical too. Yet one model repeatedly gives extreme probabilities for uncertain situations.

The analysts compare proper probability scores and inspect calibration rather than treating the identical label accuracy as the whole evaluation. They keep the operational threshold decision separate from the probability-quality assessment. The case illustrates why a probability forecast contains information that a class label discards. No real company or measured model improvement is implied.

Common misconceptions and mistakes

Proper scoring does not mean that every individual outcome rewards the true probability most. Expectation over outcomes is essential. Similarly, a better observed average on a small sample can reflect random variation and should not be treated as an infallible ranking.

Do not confuse proper with strictly proper. A proper rule allows the truthful distribution to be an optimum; strict propriety requires uniqueness. Nor should classification accuracy be assumed to reward truthful probabilities: thresholded accuracy can ignore differences between many distinct probabilities that produce the same label.

Finally, a scoring rule for full distributions is not interchangeable with every loss for a point estimate. Squared error for a point prediction targets a mean; the binary Brier construction evaluates an event probability. State exactly what is being forecast and scored.

Business use

Proper accuracy scoring rules help businesses select and monitor models whose probabilities feed several downstream decisions. Reliable uncertainty estimates can be useful when costs or decision thresholds vary among users or over time.

Keep probability evaluation separate from the value of a particular action policy. Even well-estimated probabilities need appropriate costs, constraints, and decision rules to support useful action. Reports should name the score, its direction, the evaluation population, and a baseline rather than presenting a number as universal accuracy.

Check your understanding

A forecaster reports 0.80 for an event that does not occur. What is the binary Brier loss used here, and does this one result prove the forecast was unreasonable?

Answer: The loss is (0.80 − 0)^2 = 0.64. One outcome alone does not determine whether the probability assessment was well founded.

Related terms

Brier score
Log Loss
Calibration
Scoring rule
strictly proper scoring rule
probabilistic forecast
expected loss
classification accuracy

Learn next

Repeat the expected Brier-loss calculation for a true event probability of 0.30. Compare forecasts of 0, 0.30, and 0.50, then distinguish expected score from the score after one realized outcome.

Sources

Harrell, Glossary of Statistical Terms, proper accuracy scoring rule: https://hbiostat.org/glossary/

Gneiting and Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation: https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf

The primary references ground propriety and strict propriety. Arithmetic, case, and exercise are original.

Previous: Propensity scores Next: Proper generalized decomposition