Does a large training - validation gap prove overfitting?

No. It is a signal that also requires checking data and evaluation differences.

Is a small gap always desirable?

Only alongside acceptable performance; equally poor results remain poor.

Will more training data always help?

No. The benefit depends on what currently limits performance and the relevance of the additional data.

AI Glossary / Mathematics and optimization / Advanced

Diagnosis

Definition

In this machine-learning context, diagnosis means investigating a trained model's performance by comparing training error, validation error and baselines. The comparison helps identify possible problems with fitting, generalization, data preparation or evaluation. It supports hypotheses about what to improve; a pair of error values alone does not establish the cause of a problem. This entry concerns model diagnosis rather than the medical use of the word.

What it means

A model can fit its training examples well while performing poorly on new examples. Another can perform poorly on both. These patterns suggest different investigations, but their interpretation depends on the task, measurement method and data split.
Diagnosis asks why performance looks as it does and what evidence would distinguish competing explanations. For example, a large training - validation gap might suggest overfitting, but it can also reflect a different validation population or inconsistent preprocessing.

How it works

Choose a task-relevant metric and evaluate the fitted model consistently on training and validation data. Compare both with a baseline. Check sample sizes, split design, label quality and whether preprocessing learned from training data was applied correctly elsewhere.
A validation curve varies a model setting, such as regularization strength. A learning curve varies the amount of training data. These comparisons can help distinguish whether more data, a different model or a revised training procedure is promising. Keep final test data separate from repeated development decisions; a validation score used to select a model is no longer an untouched final assessment.

Three practical examples

A text classifier makes few training errors but many validation errors. Analysts examine model complexity and regularization while also checking whether messages from the same conversation were separated appropriately across datasets.
A forecasting model performs poorly on both datasets compared with an existing baseline. The team checks optimization, useful predictors and the timing of available information before assuming that merely collecting more of the same data will solve the problem.
An image model has acceptable average validation performance but struggles on images from one camera type. Grouped error analysis helps identify a problem hidden by the overall score and motivates a focused investigation of input differences.

Think of it this way

Think of troubleshooting a machine using several gauges. One unusual reading narrows the possibilities but rarely identifies the faulty component by itself. Comparing readings, checking the measurement setup and changing one condition at a time produces a more useful diagnosis. A model's training and validation errors are gauges; they are not a complete explanation of the system.

Worked example

Consider three fictional classifiers evaluated on the same 1,000 training examples and 200 validation examples. Use classification error rate, where lower is better. The counts below are constructed to demonstrate diagnostic reasoning, not results from an actual experiment.
A simple model makes 250 training errors and 54 validation errors. Its rates are 250/1,000 = 25% and 54/200 = 27%, a two-percentage-point gap. The small gap does not establish good performance: both rates could still be poor relative to the task's requirements and baseline.
A more flexible model makes 20 training errors and 48 validation errors. Its rates are 2% and 24%, a 22-percentage-point gap. It fits the training data much better, but its validation improvement over the simple model is only three percentage points. This pattern motivates an overfitting investigation, alongside checks of the split and data preparation.
A regularized version makes 60 training errors and 24 validation errors. Its rates are 6% and 12%, with a six-percentage-point gap. Compared with the flexible version, its training error is higher but its validation error is lower by 12 percentage points, a 50% relative reduction from 24%.
The regularized model looks more promising on this validation set. These counts do not establish significance, future performance or the improvement's unique cause. Further selection uses a development protocol, followed by an independent final assessment of the chosen approach.

Illustrative case study

In a fictional delivery-planning company, a model has a large gap between training and validation performance. The team initially plans to reduce model complexity. Before changing the model, it inspects the input preparation.
Training distances were expressed in kilometers, but one validation export supplied miles under the same column name. The model was receiving inputs with a different meaning. A complexity change could not address that inconsistency directly.
The team corrects the units and reruns the evaluation with the same model and a documented transformation. It then reassesses the remaining gap rather than declaring the model fixed from the preprocessing discovery alone. The case illustrates why diagnosis combines metric comparisons with examination of the data and evaluation process. A pattern compatible with overfitting is not exclusive evidence of overfitting.

Common misconceptions and mistakes

Low training error is not sufficient evidence of generalization. Similar training and validation errors are also not sufficient: both can be unacceptably high. Conversely, validation error below training error is not automatically impossible; sampling variation, easier validation examples or different training-time behavior can contribute.
A baseline's meaning matters. An observed human or existing-system score is not automatically the irreducible error. Repeatedly inspecting final test results to choose improvements also compromises that test's role as an independent assessment. Diagnose on development data and reserve a suitable final evaluation.

Business use

Diagnosis helps prioritize model-development effort. It can reveal whether the next useful step is a data-quality correction, a revised split, a more appropriate metric or an experiment with model settings. Keep a record of observations, competing explanations and the checks used to distinguish them. Compare the chosen metric with operational requirements, because a smaller error gap does not itself demonstrate better business outcomes.

Check your understanding

What is the flexible model's error gap?

Answer: Twenty-two percentage points.

Why can the regularized model be preferable?

Answer: Its validation error is lower despite higher training error.

What explains the fictional delivery team's initial gap?

Answer: Inconsistent distance units contributed to it; the remaining performance still required reassessment.

Related terms

Learn next

Compare learning curves with validation curves and practice reading both alongside a baseline. Then design a sequence of checks that distinguishes model-fitting problems from data-preparation and evaluation problems.

Baseline

Sources

Aalto Dictionary of Machine Learning, diagnosis: comparing training and validation error with a baseline. https://aaltodictionaryofml.github.io/ADictML_Main.pdf Scikit-learn, Validation curves: plotting scores to evaluate models, sections 3.5.1–3.5.2: validation curves, learning curves and separation of selection from final evaluation. https://scikit-learn.org/stable/modules/learning_curve.html The classification-error calculations and fictional distance-unit case are original teaching illustrations.

Previous: Deviation scores Next: Diagnostic analytics