Is a holdout always a test set?

Terminology varies, so specify whether it is used for validation or final testing.

Can a forecasting holdout be randomly selected?

Sometimes a task-specific design can justify it, but ordinary future forecasting generally requires respecting time order and information availability.

Can the final model use the holdout later?

Yes, after evaluation, if the process is documented; the same observations can no longer provide a fresh untouched test of that refitted model.

AI Glossary / Time series and signal processing / Intermediate

Holdout set

Definition

A holdout set is a portion of data deliberately excluded from a specified stage of model development so that predictions can be evaluated on observations not used in that stage. In forecasting, a holdout commonly consists of later time periods. Its role must be explicit: a validation holdout supports model selection, while a final test holdout is reserved for evaluating the selected procedure.

What it means

The purpose is to approximate the challenge of predicting information that the model has not already seen. Training performance alone can be misleading because a model may adapt to quirks of the data used to build it.
Merely labeling records as holdout does not ensure an independent assessment. If their outcomes influence feature choices, parameter tuning, preprocessing, or repeated redesign, they have participated in development. The reported result must reflect how the data were actually used.

How it works

Choose a split that matches the intended prediction setting. For a future forecast, train on earlier information and evaluate on later outcomes; random mixing across time can allow information that would not have existed at issuance. Keep preprocessing estimates, feature selection, and tuning inside the development portion.
Define the evaluation horizon and whether models remain fixed or are updated as observations arrive. A rolling-origin protocol can legitimately use earlier test-period outcomes for later forecasts when that update rule was specified in advance and matches deployment. It differs from evaluating one fixed multi-step forecast issued at the split date.

Three practical examples

A demand team reserves the latest year to assess a model selected using earlier years and earlier validation windows.
A service forecaster holds out several future weeks and evaluates forecasts at the lead time needed for staffing decisions.
A broader machine-learning project holds out entire customers when deployment concerns new customers, avoiding leakage from near-duplicate records belonging to the same person.

Think of it this way

Think of rehearsing with practice questions and then taking an unseen examination. The exam is informative because the answers did not guide the preparation. If someone repeatedly studies the marked exam and revises the method to improve that same score, it becomes another practice set rather than a fresh assessment.

Worked example

Consider an original forecasting protocol with 36 monthly observations. Months 1-24 are used for initial development, months 25-32 for validation, and months 33-36 for a final test. Suppose the method is selected before opening the final test outcomes, then refitted on months 1-32 under a predeclared rule.
At the end of month 32 it issues a fixed four-month forecast: 110, 115, 120, and 125 units. The later test outcomes are 108, 119, 117, and 130. Actual-minus-forecast errors are −2, 4, −3, and 5. Mean absolute error is (2 + 4 + 3 + 5)/4 = 3.5 units. Mean squared error is (4 + 16 + 9 + 25)/4 = 13.5 squared units, and mean error is 4/4 = 1 unit.
These errors correspond to horizons one through four from one origin. They are not four independent one-step forecasts, and their average does not separately establish performance at each horizon. If the team redesigns the model after inspecting these outcomes, this set becomes development evidence; a fresh assessment is needed for an unbiased final claim. Four test observations also provide only limited evidence about general performance.

Illustrative case study

Illustrative fictional case: Spruce Retail reports excellent holdout accuracy after trying many feature combinations on the same reserved quarter. An internal review finds that the quarter's results repeatedly guided model changes.
The team relabels that quarter as validation data and evaluates the frozen procedure on a later untouched period. It records the model version and issue times so future comparisons can be reproduced. The fictional case illustrates how honest labeling preserves the meaning of an evaluation without pretending that a previously inspected dataset can become unseen again.

Common misconceptions and mistakes

A holdout is not guaranteed to represent future operating conditions. It may contain an unusual event, omit important seasons, or come from a regime that later changes. A very small holdout can also yield unstable performance estimates.
Do not compute normalization statistics, missing-value replacements, or feature rankings from the full dataset before splitting if those steps learn information from future records. Leakage can occur outside model fitting. Repeatedly selecting the best model on the final test score also creates selection bias, even when no test row is directly included in a training function.

Business use

A credible holdout evaluation helps businesses compare forecasting procedures before committing resources. Its usefulness depends on matching the evaluation to the decision, data availability, and intended update policy.
Document split dates, grouping rules, target horizons, model-selection steps, and which results were visible to developers. Report performance alongside benchmark results and evidence limitations. After final evaluation, retraining on additional data may be appropriate for deployment, but it produces a revised fitted model whose future behavior still requires monitoring.

Check your understanding

In the example, why can months 25-32 be used in the final refit?

Answer: They were validation data, and the refit rule was fixed before examining months 33-36.

Why is mean error smaller than mean absolute error?

Answer: Positive and negative errors cancel in mean error.

Would trying ten more models on months 33-36 preserve their final-test role?

Answer: No. Their outcomes would guide selection.

Related terms

Learn next

draw a timeline of your intended evaluation, marking when every feature and outcome becomes available. Compare a fixed-origin holdout with rolling-origin validation, then choose a final test period that supports the actual decision horizon.

Sources

Sources: Hyndman and Athanasopoulos, Forecasting: Principles and Practice, training/test sets and forecast accuracy: https://otexts.com/fpp3/toolbox.html . scikit-learn, official Common pitfalls and recommended practices, data leakage: https://scikit-learn.org/stable/common_pitfalls.html . Primary textbook and documentation consulted. The split, calculations, and fictional case are original illustrations.

Previous: Holdout data Next: Holm - Bonferroni method