Is back-testing the same as cross-validation?

They overlap. Rolling-origin cross-validation is one structured approach to evaluating historical forecasts; back-testing is a broader term.

Can random splitting work for time-dependent forecasting?

It often misrepresents the available information. Use a design justified by the task and temporal dependencies.

Should the model be retrained at every origin?

Only if that represents the deployment or comparison being evaluated. State the update schedule.

AI Glossary / Governance, ethics and safety / Intermediate

Back-testing

Definition

Back-testing evaluates a model, forecasting method or decision procedure using historical observations to examine how it would have performed. A credible test recreates the information available at each past decision point and compares outputs with outcomes that became known later.

What it means

A demand model may fit last year's records well because it has already learned from them. That is not the same as showing that it could have forecast each week before that week's demand was known. Back-testing tries to recreate the latter question.

The term is widely used in forecasting and other model-validation settings. Its exact metrics depend on the application: prediction error, interval coverage or operational outcomes may be relevant. This entry uses business forecasting examples rather than treating a historical result as an investment recommendation.

A back-test is evidence about specified past conditions. It does not guarantee future performance, especially when products, behavior, data collection or operating rules change. Its value depends heavily on whether the historical simulation resembles the actual deployment process.

How it works

First define the decision time, forecast horizon, training history and outcome measure. At each historical origin, fit or update the model using only information then available, generate the required prediction and compare it with the later observation. Move the origin forward and repeat.

An expanding window retains earlier training observations as new ones arrive; a rolling window uses a fixed recent span. The choice should reflect the proposed operating process. Preprocessing, feature selection and tuning must also respect the time boundary. Fitting those steps on the complete dataset can leak future information even if the final model uses a chronological split.

Keep records of data-release timing and revisions. A final historical database may contain corrected values unavailable at the original date. Compare results against a relevant baseline and examine variation across periods. If repeated test results guide model selection, reserve separate evidence for the final assessment.

Three practical examples

A retailer evaluates next-week demand forecasts by repeatedly training on earlier weeks and comparing each forecast with the following week's realized demand.
A staffing team checks whether forecast intervals cover observed enquiry volumes at the planning horizon it actually uses.
An equipment team recreates historical alert decisions using only sensor readings available before each alert, avoiding later maintenance notes as input.

Think of it this way

Think of replaying an old match while covering the scoreboard and revealing events only as they originally happened. A prediction made with the final score visible would not demonstrate foresight.

The analogy has limits: business records may be revised after the event, and a deployed decision can change what happens next. Back-testing therefore needs explicit assumptions about both information availability and whether historical outcomes remain a valid comparison.

Worked example

Evaluate the forecast you could have made

Suppose an invented service forecasts weekly enquiry counts. At the end of week 4, it uses weeks 1-4 to predict week 5 as 110; the actual count is 120. At the next origin, it uses weeks 1-5 to predict week 6 as 125; actual demand is 115. Its week-7 prediction is 130 and actual demand is 135.

Absolute errors are 10, 10 and 5. The mean absolute error across these three forecasts is 25/3, approximately 8.33 enquiries. A baseline evaluated on those same weeks must use the same information boundary and horizon.

If the model instead used a corrected customer-count feature published after week 7 for all three forecasts, the calculation would still be arithmetically correct but the evaluation would no longer represent the original forecasting opportunity. Three observations also provide very limited evidence about seasonal performance.

Illustrative case study

Fictional Alder Catering builds a model for next-month order volumes. Its initial evaluation randomly mixes months into training and testing and reports strong performance. A review finds that later revisions to customer contracts reveal information about earlier demand.

The team reconstructs which contract version was available at each month-end and uses rolling forecast origins. It evaluates the actual one-month planning horizon and compares the model with a seasonal baseline. It also reports months with unusual closures separately rather than concealing them in an average.

The revised process gives a more relevant account of historical performance. The company still pilots the model prospectively because future conditions and the operational effects of staffing decisions cannot be fully established by replaying records.

Common misconceptions and mistakes

Good training fit is a good back-test. Training fit uses observations that influenced the model.
A chronological final split prevents all leakage. Earlier preprocessing and later data revisions can still introduce unavailable information.
Testing many alternatives leaves the test untouched. Repeatedly choosing the winner using the same period adapts decisions to that period.
Past success guarantees future success. Changing conditions and imperfect simulation limit that inference.

Business use

Back-testing helps businesses compare methods before relying on them operationally. Document the decision that would have been made, the information then available and the consequences being measured. Match the forecast horizon to the lead time of the real decision.

For action-taking systems, historical replay may miss feedback: a different price, staffing level or recommendation could change the outcome. Make those assumptions visible and use prospective evaluation where appropriate. Preserve the evaluation configuration and data versions so reviewers can distinguish a reproducible historical test from an attractive chart produced after extensive unreported experimentation.

Check your understanding

A team predicts last March using a feature that was first published in April. Does excluding March's target from training make this a realistic March forecast test?

Answer: No. The feature was unavailable at the decision time, so the test contains future information.

Related terms

Data leakage
Forecast horizon
Holdout set
Baseline
Concept drift
Time series cross-validation
rolling window

Learn next

Read data leakage and forecast horizon. Draw a timeline showing when each input, prediction and outcome becomes available. Then design a historical test that respects those timestamps, including any data revisions.

Sources

Hyndman and Athanasopoulos, Forecasting: Principles and Practice, Time series cross-validation: https://otexts.com/fpp3/tscv.html. NIST AI glossary context: https://airc.nist.gov/glossary/. The forecasting source supports rolling-origin evaluation; numerical and business scenarios are original teaching examples.

Previous: Back-propagation through time Next: Backbone