Is an 80/20 split always best?

No. The right allocation depends on data volume, model learning needs, rare outcomes, and evaluation precision.

Can repeated holdout reuse examples?

Yes. Overlap across repetitions is typical, unlike a single complete k-fold cycle's disjoint held-out folds.

Should a final test set be used repeatedly during tuning?

Repeated use can influence choices and make the reported result optimistic; a distinct validation process helps preserve the final test's role.

AI Glossary / Causal inference and experimentation / Intermediate

Holdout cross-validation

Definition

Holdout cross-validation is an imprecise phrase commonly used for evaluating a model on a held-out portion of data, sometimes repeated across several random splits. A single holdout divides data once into training and evaluation subsets. Repeated holdout performs several such splits and fits a fresh model for each. These differ from k-fold cross-validation, where predefined folds take turns being held out. The exact splitting procedure should be stated rather than inferred from the phrase alone.

What it means

The central idea is to assess predictions on examples that did not participate in the corresponding model fit. A single split is simple and can be appropriate when enough representative data is available. Repeating the split shows how results vary with the partition, but the evaluation subsets may overlap and some observations may be tested more often than others. Neither approach automatically protects against related records, time leakage, or repeated tuning based on evaluation results.

How it works

Choose the evaluation question and split unit, then allocate data to training and held-out assessment. Fit preprocessing and the model only on the training portion. Apply the fitted process to the held-out portion and calculate the declared metrics. For repeated holdout, create new splits under the same rules, refit from scratch, and summarize the resulting assessments while accounting for overlap. Use grouping or time ordering when the deployment setting requires it. Keep model selection separate from a final test assessment if the held-out results guide development choices.

Three practical examples

Large tabular dataset: reserve a representative subset for assessment while fitting the model on the remaining data.
Development comparison: use repeated random splits to investigate sensitivity to which suitable independent examples enter training.
New-customer prediction: hold out entire customers, not random transactions from customers already represented in training.

Think of it this way

Think of setting aside a sealed practice packet before preparing for an exam. The packet provides a check only while its contents remain outside preparation. Repeatedly choosing packets can reveal variation, but overlapping questions do not become new independent evidence each time. The analogy also shows why repeatedly studying the held-out answers weakens their role as an unseen assessment.

Worked example

Suppose 1,000 suitable independent records are randomly divided into 800 training records and 200 held-out records. A fitted classifier gets 174 held-out labels correct, giving 174/200 = 87% accuracy. That is one split's estimate, not a guarantee that every future batch will achieve 87%.
Now repeat an 80/20 random holdout procedure five times, obtaining accuracies of 85%, 87%, 86%, 88%, and 84%. Their mean is 86%. Although there are five times 200 = 1,000 evaluation slots, they are not necessarily 1,000 distinct test records. A record may appear in multiple held-out subsets and others in none. If each split independently selects 20% of records for evaluation, the probability a particular record is never held out is 0.8 to the fifth power = 0.32768. The expected number of distinct records evaluated at least once is therefore 1,000 times (1 - 0.32768), or about 672.32. This illustrates overlap, not a required sample-size target or a claim that the repeated scores are independent.

Illustrative case study

A fictional demand team reports strong results from a random holdout, then realizes the model is intended to predict future periods. The random split had allowed later observations into training when assessing earlier ones. The team switches to an evaluation design that trains on earlier data and tests on later data, including realistic feature availability. It retains the old score only as a clearly labeled historical experiment. The corrected split better answers the deployment question, even though its score is less flattering.

Common misconceptions and mistakes

A single holdout is not the same as k-fold cross-validation. Repeated holdout does not guarantee every example is evaluated exactly once, and repeated scores are not automatically independent. Stratification preserves class proportions but does not solve grouping or temporal leakage. A random seed makes a split reproducible, not representative by itself. Using held-out scores to choose models turns that subset into validation data; calling it a test set does not restore independence from the development process.

Business use

Use holdout methods when their simplicity and computational cost fit the available data and decision. Report split fractions, sample counts, seed or reproducibility method, grouping, stratification, and temporal rules. Fit all learned transformations within the training data for each split. Inspect performance across important case types and retain enough positive or rare-class examples for meaningful assessment. When comparing candidates, use a paired design with the same splits where appropriate, then protect a separate final assessment from repeated selection decisions.

Check your understanding

Why do five 200-record held-out evaluations not necessarily test 1,000 different records?

Answer: The randomly selected subsets can overlap. Repeated evaluation slots count appearances, while unique tested records count each record only once.

Related terms

Holdout set
k-fold cross validation
Validation set
Test set
Data leakage
train-test split
repeated holdout
random subsampling validation
grouped split
temporal split

Learn next

Draw one train-holdout split and record which steps learn from which portion. Then draw several repeated splits and count unique held-out records separately from total appearances. Compare this with k-fold assignment. Study grouped and chronological evaluation next so the splitting method reflects the kind of unseen case the model will actually encounter.

Sources

Scikit-learn, cross-validation and random permutations: https://scikit-learn.org/stable/modules/cross_validation.html Scikit-learn, ShuffleSplit: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html Scikit-learn, train_test_split: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html Scikit-learn, common pitfalls: https://scikit-learn.org/stable/common_pitfalls.html

Previous: HOG feature Next: Holdout data