Must the evaluation wait for genuinely new data?

Not always. A historical holdout can simulate a later period if the model and all inputs respect the earlier information boundary.

Is post-sample evaluation identical to any out-of-sample test?

It is a time-oriented forecasting form of out-of-sample assessment. A random holdout does not necessarily have that temporal meaning.

Can the model be updated during evaluation?

Yes, if updates follow a declared schedule and use only information available at each new forecast origin.

AI Glossary / Causal inference and experimentation / Intermediate

Post-sample evaluation

Definition

Post-sample evaluation assesses a forecasting model using observations from after the period on which it was estimated. It compares forecasts with later outcomes that were unavailable to the model when those forecasts were made. A historical end-of-series holdout can support such an assessment, and ongoing tracking of newly arriving outcomes is another form. The essential idea is to evaluate forecasting beyond the estimation sample rather than judging only how closely a model fits past data.

What it means

A close historical fit need not yield accurate forecasts. Post-sample evaluation asks how well earlier information predicted what came next.

Randomly dividing a time series can allow later patterns or related observations to influence a model assessed on earlier periods. That does not reproduce an ordinary forward-looking forecasting task.

Post-sample results are still conditional evidence. An evaluation period may be unusually quiet, unusually volatile, or unlike the intended operating conditions. Report which period and forecasting horizon were assessed rather than treating one score as a timeless description of the model.

How it works

Choose a forecast origin: the point up to which information is available. Fit the model using the permitted earlier data, then generate forecasts for specified later periods. Preserve those predictions before comparing them with the observed outcomes.

Calculate errors using a declared convention, such as actual minus forecast, and summarize them with suitable measures. Compare with a simple benchmark evaluated on the same periods and horizons. For repeated assessment, move the origin forward and refit according to the intended update schedule, using only information available at each origin.

Account for predictors and data revisions too. A forecast using the actual future value of a predictor that was unknown at the origin is not equivalent to the intended live forecast. Distinguish a fixed-origin multi-step assessment from a sequence of updated one-step forecasts because they answer different questions.

Three practical examples

1. Store demand: a retailer estimates a model through September and assesses forecasts for October through December against the later observed sales.

2. Service workload: an operations team records each week's forecast before the next week begins, then tracks its errors as actual ticket counts arrive.

3. Energy planning: analysts compare models over several historical forecast origins, reproducing the information and update schedule that would have been available at each point.

Think of it this way

Think of writing tomorrow's weather prediction in a sealed envelope and opening it after tomorrow has passed. The timing makes it possible to distinguish prediction from hindsight. The analogy is limited because real forecasting also involves model selection, revised datasets, changing predictors, and different horizons. Merely saving an output is insufficient if future information already influenced its construction.

Worked example

Suppose a fictional business fits a demand model using months 1 through 9. At the end of month 9, it forecasts months 10, 11, and 12 as 110, 120, and 130 units. The later actual outcomes are 100, 125, and 145.

Using actual minus forecast, the errors are −10, 5, and 15. Mean absolute error is (10 + 5 + 15)/3 = 10 units. Mean error is (−10 + 5 + 15)/3, approximately 3.33 units, indicating net underprediction across this small set under the chosen sign convention.

A benchmark that forecasts 100 units for all three months has absolute errors 0, 25, and 45, giving mean absolute error 70/3, approximately 23.33. The candidate performs better by this measure on these three observations. The result is a fixed-origin three-month assessment; it does not show how a model refitted each month would perform, nor does such a short period establish reliable future superiority.

Illustrative case study

Fictional case: Alder Catering evaluates weekly demand forecasts using a year-end dataset containing corrected event attendance counts. The team realizes that some corrected counts were published only after the forecast dates.

It reconstructs the information that would actually have been available at each origin and reruns the assessment. It also separates one-week-ahead results from forecasts made four weeks ahead, because staffing decisions use both horizons. The case illustrates why data availability and horizon must match the decision process. It does not report a real company's scores or a guaranteed benefit from any model.

Common misconceptions and mistakes

Residuals from the fitting period are not the same as post-sample forecast errors. A model's parameters were chosen using the fitting sample, so close historical fit alone is insufficient evidence of forecasting ability.

Do not repeatedly tune a model to one holdout period and continue presenting that period as untouched final evidence. It has become part of development. Use an appropriate additional assessment or state the selection role clearly.

Other mistakes include using future predictor values, comparing models on different dates, and mixing horizons without explanation. Missing outcomes and exceptional events also need a consistent policy. Excluding difficult periods after seeing their errors can make the reported score unrepresentative of the actual forecasting task.

Business use

Post-sample evaluation helps businesses decide whether a forecasting model improves planning relative to a simpler alternative. It makes the evidence more relevant to decisions made before outcomes are known.

Retain forecast timestamps, model versions, available inputs, horizons, and later outcomes. Report errors in units that planners understand and examine periods that matter operationally. If decisions depend on uncertainty estimates or asymmetric costs, evaluate those aspects as well as point-forecast accuracy. A lower average error is useful evidence, but the business consequence depends on how forecasts are used.

Check your understanding

A forecast for month 12 was made at the end of month 9. An analyst replaces it after seeing month 11's outcome and calls it the same three-step-ahead forecast. Is that accurate?

Answer: No. The information set and origin changed. The updated forecast may be valid, but it must be labeled and evaluated as a different forecasting exercise.

Related terms

Holdout set
Forecast horizon
Residual
Data leakage
forecast origin
rolling-origin evaluation
forecast error
backtesting

Learn next

Create a small monthly series and reserve its last three observations. Record one fixed-origin forecast for all three, then design a separate rolling one-step procedure. Explain which information each prediction is allowed to use.

Sources

Glossary of Forecasting Terms, Post-sample evaluation, author-hosted reference: https://robjhyndman.com/mwh3/FG4.pdf

Hyndman and Athanasopoulos, Forecasting: Principles and Practice, evaluation and time-series cross-validation: https://otexts.com/fpp3/toolbox.html

These references support the temporal definition and evaluation design. Numbers, case, and exercise are original illustrations.

Previous: Post-processing algorithm Next: Post-trained model