Can a forecast have negative skill?

Yes. It performed worse than the selected reference under the selected score.

Does zero skill mean no useful information of any kind?

No. It means no improvement under that particular comparison.

Is the strongest possible benchmark always required?

The benchmark should be credible and explicit; reporting several meaningful references can clarify what the model adds.

AI Glossary / Statistics and probability / Intermediate

Forecast skill

Definition

Forecast skill describes how well a forecasting system predicts compared with a specified reference or benchmark, under a defined evaluation method. A skill score often expresses relative improvement over that reference. For a nonnegative loss where zero is perfect, a common form is one minus forecast loss divided by reference loss. Positive values indicate improvement, zero indicates equal loss and negative values indicate worse performance. The term is also used more broadly in forecast verification, so the exact score, reference and evaluation sample should always be stated.

What it means

Raw error alone does not reveal whether a forecasting system adds value over an easy alternative. A small error may still be worse than predicting the last observed value in a stable series. Conversely, a difficult series may have larger absolute errors while a model substantially improves on a sensible baseline. Skill is therefore relative to a task, information set and benchmark. It is not an intrinsic permanent property of an algorithm, and it does not automatically translate into economic benefit under every decision rule.

How it works

Specify the forecast target, horizon, evaluation cases and loss function. Generate both model and reference forecasts using only information available at each forecast origin. Score them against the same outcomes with consistent weights and missing-data rules. Aggregate each loss over the evaluation set, then compute the chosen skill score. For zero-optimal loss L, SS = 1 - L_model/L_reference requires a positive reference loss. Other scores may have different normalizations. Examine uncertainty, performance by horizon and relevant subgroups, and sensitivity to the reference choice. Use proper scoring rules for probabilistic forecasts when the aim is to evaluate the reported predictive distribution honestly.

Three practical examples

Demand forecasting: compare a model with a seasonal-naive benchmark at the actual replenishment horizon.
Event prediction: assess probability forecasts against a historical-frequency reference using Brier skill score.
Energy planning: compare competing forecast systems over identical dates, horizons and locations before attributing differences to model quality.

Think of it this way

Think of timing a runner against a clearly stated reference performance on the same course. A faster time shows improvement relative to that reference. Change the course, conditions or reference runner and the comparison changes; the number cannot be interpreted in isolation.

Worked example

Consider four hypothetical binary outcomes [1, 0, 1, 0] and probability forecasts [0.8, 0.3, 0.7, 0.2]. Their squared probability errors are 0.04, 0.09, 0.09 and 0.04, so the Brier score is 0.26/4 = 0.065. Suppose a reference forecast, fixed in advance, assigns probability 0.5 to every event. Its Brier score is 0.25. The Brier skill score is 1 - 0.065/0.25 = 0.74, meaning a 74% reduction in this average squared-probability error relative to that reference on these four cases. It does not mean 74% of forecasts were correct. If another legitimate reference had score 0.05 on the same cases, the model's skill relative to it would be 1 - 0.065/0.05 = -0.30. The small example demonstrates benchmark dependence, not reliable evidence of future superiority.

Illustrative case study

Fictional case: Acorn Forecast Lab evaluates an AI model against an existing seasonal baseline. The model improves average error at short horizons but underperforms for a particular location at longer horizons. Analysts publish both results and retain the baseline where it performs better. They also examine operational costs before deciding how to use the forecasts. This invented case illustrates why an overall skill result should not be stretched into a claim that a model is best everywhere or that it necessarily improves every business decision.

Common misconceptions and mistakes

Do not interpret a skill score as classification accuracy or a probability that the model is correct. A negative value is possible and meaningful. Do not compare scores computed against different references without explaining the difference. Reference forecasts estimated using future information create an unfair or misleading evaluation. A perfect reference with zero loss makes the simple ratio undefined. Averaging casewise ratios is generally not equivalent to the ratio of aggregate losses. High correlation can coexist with large bias, and improvement on one scoring rule does not guarantee improvement on another with a different target.

Business use

Businesses can use skill comparisons to test whether forecasting complexity earns its place. Choose a credible baseline that reflects a practical alternative, retain forecast timestamps and evaluate the horizons relevant to decisions. Report the underlying losses alongside the relative score so users can see the absolute scale of improvement. Assess uncertainty using methods that respect temporal or spatial dependence. Keep business utility separate from statistical skill, because costs, constraints and action thresholds influence the value of a forecast. Reevaluate after changes in the environment rather than assuming historical relative performance will persist.

Check your understanding

Model loss is eight and reference loss is 10, with zero loss perfect. What is the relative skill score?

Answer: 1 - 8/10 = 0.20, a 20% reduction in that loss on the evaluated cases. It is not an 80% or 20% classification-accuracy statement.

Related terms

Benchmarks
Brier score
Calibration
Forecast horizon
Forecast accuracy
benchmark
Brier skill score
proper scoring rule
rolling-origin evaluation
economic value

Learn next

Learn how scoring rules align with point and probability forecasts. Then compare naive, seasonal and climatological references, and study uncertainty in paired forecast evaluations before interpreting small differences in skill.

Sources

ECMWF, Statistical Concepts, probabilistic verification and Brier skill score: https://confluence.ecmwf.int/pages/viewpage.action?pageId=283544832 ECMWF, Forecast evaluation: https://www.ecmwf.int/en/research/modelling-and-prediction/forecast-evaluation Gneiting, Making and Evaluating Point Forecasts: https://arxiv.org/abs/0912.0902

Previous: Forecast horizon Next: Forecasting