Is a smaller gap always better?

No. A model with equally poor training and held-out performance can have a tiny gap.

Is test error the generalization gap?

No. The gap is a difference from training performance; test error is one performance estimate.

Can the gap change without retraining?

Yes. Evaluating the same fitted model on another sample or population can change the estimated difference.

AI Glossary / Mathematics and optimization / Intermediate

Generalization gap

Definition

The generalization gap is the difference between a model's performance on its training data and its performance beyond those data. Using loss, a common convention is expected loss on new observations minus average training loss. Since the true data-generating distribution is usually unknown, practitioners estimate the difference with held-out data. The sign convention and metric should always be stated, because accuracy and loss are interpreted differently.

What it means

Training performance measures how well the model fits information it has already used. Generalization concerns new observations relevant to the intended task. A model can fit its training set unusually well because it has adapted to patterns or noise specific to that set. The gap helps reveal that discrepancy, but it is not the same as generalization error itself. A model can have a small gap because both training and new-data performance are poor. The goal is useful performance on relevant new data, not merely making two scores equal.

How it works

For a fixed fitted model h, define its training loss as the average of the chosen loss over the training records. Define population risk as the expectation of that same loss under the target distribution. Their difference is the loss-based generalization gap. A separate sample can estimate population risk when its sampling and evaluation conditions are appropriate. The resulting held-out difference is an estimate with sampling variability. Repeatedly choosing models based on that same held-out sample can make its score optimistic, so model selection and final evaluation need to be distinguished.

Three practical examples

A classification team can compare training and held-out error rates after fitting, using the same label definitions and prediction procedure.
An analyst can plot training and validation losses against training-set size to investigate whether additional representative data may reduce a persistent discrepancy.
A model review can compare candidate systems using their actual held-out losses as well as their gaps, avoiding selection of a uniformly poor model simply because its scores match.

Think of it this way

Think of practicing with one set of exercises and then attempting fresh exercises drawn from the intended subject. Strong practice results do not automatically transfer. The analogy depends on the fresh exercises being relevant and fairly selected; a much harder exam or a different subject introduces a change in conditions, not just evidence of memorizing the practice set.

Worked example

Suppose a classifier makes 20 errors on 1,000 training examples and 30 errors on 200 independent held-out examples. Training error is 20/1,000 = 0.02, and held-out error is 30/200 = 0.15. The estimated loss-based gap is 0.15 − 0.02 = 0.13, or 13 percentage points. Training accuracy is 98% and held-out accuracy is 85%; defining an accuracy gap as training accuracy minus held-out accuracy gives the same 13-point difference. It is not a 13% relative increase in error: the held-out error rate is 7.5 times the training rate. Now compare a second model with training error 0.20 and held-out error 0.21. Its estimated gap is only one percentage point, but its held-out performance is worse than the first model's 0.15. Shrinking the gap alone would therefore rank these models incorrectly for an error-minimization goal. The 0.15 held-out rate is also an estimate, not an exact population risk. A different sample of 200 observations can produce a different count, and evidence about one population does not automatically extend to another.

Illustrative case study

In a fictional support-ticket project, Hazel Analytics observes excellent training accuracy and a lower validation score. The team checks for duplicate tickets, inconsistent preprocessing and time-related changes before concluding that model complexity is the only issue. It compares revised candidates using a defined development procedure and reserves another evaluation set for the final choice. The case illustrates investigation of a discrepancy; it does not claim that a particular change removes overfitting or delivers a measured operational gain.

Common misconceptions and mistakes

A large observed gap can be consistent with overfitting, but differences in data distribution, preprocessing or evaluation settings can also matter. A zero gap does not establish accuracy, fairness or robustness. A negative estimated loss gap is possible through sampling variation or different evaluation conditions; it is not automatically an error. Comparing regularized training objectives with unregularized test loss can create an artificial mismatch. Data leakage can make a gap look reassuringly small while invalidating the evaluation. Finally, percentage points and relative percentages are different quantities and should not be interchanged.

Business use

Report both scores, sample sizes, metric definitions and split procedure. Evaluate training and held-out records under comparable prediction settings, and keep preprocessing fitted within the proper training boundary. Choose splits that match the deployment question, such as separating future periods or related entities where appropriate. Investigate uncertainty and subgroup behavior rather than relying on one aggregate difference. Use validation to guide development and protect the independence of final evaluation. If the deployment population changes, describe that shift explicitly rather than treating the original estimated gap as a permanent model property.

Check your understanding

Which worked model has lower held-out error: the one with a 13-point gap or the one with a one-point gap?

Answer: The first, with held-out error 0.15 rather than 0.21. Gap size alone does not determine predictive quality.

Related terms

Learn next

calculate error and accuracy gaps from several confusion-matrix summaries, explicitly choosing the subtraction order. Then compare two models with similar gaps but different held-out errors. Design a split that answers the actual deployment question and identify how repeated model selection could bias its evaluation.

Sources

Sources: Jung and colleagues, The Aalto Dictionary of Machine Learning, generalization gap, https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; scikit-learn, Validation curves and learning curves, https://scikit-learn.org/stable/modules/learning_curve.html . Error counts, model comparison and the fictional support-ticket investigation are original; held-out differences are identified as estimates rather than exact population gaps.

Previous: Generalization curve Next: Generalization hierarchy