Is PRESS the same as leave-one-out mean squared error?

They contain the same squared residuals, but PRESS is their sum; dividing by the number of observations gives the mean.

Can PRESS exceed training error?

Yes. Omitted-observation prediction is a different and often harder assessment than fitting those observations directly.

Does it work for time-series forecasting unchanged?

Not necessarily. Forecasting usually needs a validation scheme that respects time order and the information available at prediction time.

AI Glossary / AI and machine learning / Advanced

PRESS statistic

Definition

The PRESS statistic, commonly expanded as prediction or predicted residual sum of squares, sums squared prediction errors obtained by leaving each observation out of model fitting and predicting it from the remaining observations. It is a measure associated with leave-one-out predictive assessment.

What it means

A model can fit the data used to build it well while predicting new observations poorly. PRESS asks how the model predicts each observation when that observation is excluded from its own fit.

Lower PRESS indicates smaller squared leave-one-out errors when comparing models on the same response scale and evaluation observations. It is not a universal quality score that can be compared indiscriminately across datasets, units or sample sizes. The validity of the assessment also depends on whether leaving out individual rows matches the intended prediction problem.

How it works

For each observation, fit the specified model without that observation, predict its response and calculate the residual. Square those residuals and add them. Dividing by the number of observations gives the corresponding mean squared leave-one-out error.

For ordinary least squares with an appropriate fixed design, the deleted-observation residual can be computed as e(i)/(1 − h(i)), using the full-fit residual and leverage, provided h(i) is below one and the deletion fit remains identifiable. This shortcut is not a license to reuse preprocessing learned from all data. Any feature selection or tuning involved in the assessment must be handled consistently with the validation design.

Three practical examples

1. An analyst can compare two fixed regression specifications on the same dataset using their leave-one-out prediction errors alongside other diagnostics.

2. A teaching exercise can show how a model with tiny training residuals still performs poorly when individual observations are omitted.

3. A technical reviewer can check whether a reported prediction statistic actually evaluates the full modeling workflow or only its final fitted equation. The distinction matters when earlier choices used information from the held-out cases.

Think of it this way

Imagine practicing for a quiz by repeatedly setting aside one question, studying the others and answering the omitted question. Adding up the squared mistakes gives a picture of how well that procedure handles omitted material.

The analogy also reveals a limitation: if the questions are near-duplicates or reveal one another's answers, leaving out just one may not provide a realistic test of genuinely new material.

Worked example

Suppose four leave-one-out residuals are 1, −2, 0.5 and −0.5. PRESS is 1² + (−2)² + 0.5² + (−0.5)² = 1 + 4 + 0.25 + 0.25 = 5.5.

The corresponding mean squared error is 5.5 / 4 = 1.375. These are different summaries: PRESS is a sum, while the mean adjusts for the number of predictions.

For a separate ordinary-least-squares illustration, a full-fit residual of 1 with leverage 0.5 gives a deleted residual of 1/(1 − 0.5) = 2. Its squared contribution becomes 4. This shows why a seemingly modest fitted residual can conceal a larger omitted-observation error.

Illustrative case study

Fictional service-planning team QueuePoint compares two regressions for handling time. A more complex model improves training fit but has a larger PRESS value on the same observations. The team investigates influential cases rather than automatically selecting the more complex equation.

It then notices that several rows come from the same customer interaction. Leaving out individual rows still leaves closely related information in training. The researchers add a grouped evaluation aligned with future independent interactions. PRESS provided useful evidence, but the validation unit needed to match the real prediction setting.

Common misconceptions and mistakes

PRESS is not the ordinary training residual sum of squares. The predictions must reflect omission of the evaluated observation. It also does not directly measure causal validity or explain why a model succeeds.

Avoid comparing raw PRESS across different response units or observation counts without context. Do not apply the ordinary-least-squares shortcut blindly to arbitrary models or changing feature-selection procedures. For time series, repeated measurements or clustered records, leaving out one row can leak related information. Finally, repeatedly choosing models using one validation score can itself create selection bias; a separate final evaluation may still be needed.

Business use

PRESS can help assess regression models when the leave-one-out setup fits the business question. It is useful evidence about predictive behavior, especially when considered with residual patterns, uncertainty and a relevant baseline.

Ask what was left out, which steps were repeated and whether the evaluation cases resemble future use. Report the response units and sample size, and explain the magnitude of errors in operational terms. A smaller number is only useful when the comparison is fair and the resulting improvement matters to the decision being supported.

Check your understanding

Two models have PRESS values measured on different response scales. Can the smaller number alone establish the better predictor?

Answer: No. Scale, observations and validation design must be comparable. Interpret predictive error in the context of the actual task, not as a unit-free ranking.

Related terms

Learn next

Compute the four-residual example by hand, then compare training and leave-one-out errors for a small regression. Identify whether the data contains groups or time dependence that require a different validation split.

Residual

Sources

statsmodels official OLSInfluence documentation, including PRESS residuals and their error sum of squares: https://www.statsmodels.org/stable/generated/statsmodels.stats.outliers_influence.OLSInfluence.html. NIST regression documentation: https://itl.nist.gov/div898/software/dataplot/refman1/auxillar/line_fit.htm. All numerical calculations and the QueuePoint case are original illustrations. The statsmodels PRESS documentation and NIST deleted-residual definitions were consulted.

Previous: Presequences Next: Pretokenization