Is cross-validation the same as training one model repeatedly for more epochs?

No. It involves separate fits with different held-out data, not continuing training on one model.

Must data always be shuffled?

No. Shuffling may help suitable independent data but can invalidate chronological or grouped evaluation.

Can the final model use all development data?

Yes, after choices are finalized, but its final assessment should come from evidence that was not improperly used to choose it.

AI Glossary / Causal inference and experimentation / Intermediate

Cross-validation

Definition

Cross-validation evaluates a learning procedure by repeatedly fitting it on one part of a dataset and assessing it on a separate part. The most familiar form, k-fold cross-validation, divides data into k folds and holds out each fold in turn. It provides evidence about performance on unseen examples under the chosen splitting assumptions. It is not a guarantee of future performance, and its usefulness depends on keeping information from the evaluation portion out of model fitting and selection.

What it means

A model can perform well on examples it has already seen without learning patterns that generalize. Cross-validation creates repeated opportunities to test on examples withheld from a particular fit. The evaluated object is the training procedure, including preprocessing and settings, rather than one permanently fixed fitted model. After assessment, a final model may be trained on more data. Its performance is related to, but not exactly identical with, the performance of the smaller fold-trained models.

How it works

Choose a splitting design that reflects how the model will be used. For ordinary k-fold evaluation, partition suitable observations into k groups. For each group, start with a fresh model, fit preprocessing and the model on the other groups, and score the held-out group. Aggregate results with a metric-appropriate method and inspect variation and failure patterns. Related records may need grouped splits; time-dependent forecasting needs chronological designs. If the same process selects hyperparameters, use a separate test set or an outer evaluation loop to assess selection fairly.

Three practical examples

Tabular prediction: compare a small set of candidate learning procedures with preprocessing fitted separately inside each fold.
Repeated-customer data: hold out whole customers when the goal is to predict for previously unseen customers.
Forecasting: use earlier periods for training and later periods for assessment, respecting which information would have been available at prediction time.

Think of it this way

Think of rehearsing with several practice sets while keeping a different set unseen for each assessment. The rotation gives a broader view than one split, but only if the unseen set has not leaked into preparation. The analogy stops where rehearsals are treated as independent experiments: fold training sets overlap, so their scores are statistically related.

Worked example

Suppose 120 independent labeled examples are divided into four equal folds of 30. Each run trains on 90 examples and evaluates on the remaining 30. The four models correctly classify 24, 25, 23, and 26 held-out examples. Fold accuracies are 80%, 83.33%, 76.67%, and 86.67%. Because fold sizes are equal, their average is 98/120, approximately 81.67%, which also equals accuracy from all 120 held-out predictions pooled together.
Each example appears in one evaluation fold and three training sets. That does not create 480 independent test examples. Before each fit, a numerical scaler must learn its means and scales from that run's 90 training examples, then apply them to its 30 evaluation examples. Fitting the scaler once on all 120 uses held-out information. Finally, choosing the best of many models from these same scores introduces selection effects; an untouched test set or outer cross-validation loop is needed for a less optimistic assessment of the selected procedure.

Illustrative case study

A fictional maintenance team initially splits individual sensor rows at random. Rows from the same machine appear in training and evaluation, allowing machine-specific patterns to make results look encouraging. The intended deployment involves new machines, so the team repeats evaluation with machine-level grouping. Performance falls, revealing a more realistic limitation. It then improves features and reports the grouped design. Cross-validation was useful only after the split matched the deployment question; increasing the fold count alone would not have fixed the mismatch.

Common misconceptions and mistakes

More folds do not automatically produce a trustworthy result. Leakage, temporal reversal, and related observations can undermine any fold count. A simple standard deviation across folds is not automatically a valid confidence interval because folds share training data. Averaging nonlinear metrics such as F1 can differ from calculating the metric on pooled predictions. Cross-validation also does not cure biased labels or a dataset that omits important future cases. Repeatedly inspecting results can gradually turn evaluation data into development data.

Business use

Use cross-validation when data is limited and repeated held-out assessment helps compare learning procedures. Document the split unit, grouping, time ordering, metric aggregation, and all fitted preprocessing steps. Keep a record of how model choices were made. Match evaluation to the actual question: new records from known entities, new entities, or future periods can require different designs. Report important error slices and uncertainty with appropriate methods, then retain a final validation strategy that remains meaningful after model selection.

Check your understanding

Why should feature scaling be fitted inside each training fold?

Answer: The scaling parameters are learned from data. Using the held-out fold to estimate them lets evaluation information influence the procedure being assessed, so the test no longer faithfully represents unseen-data processing.

Related terms

k-fold cross validation
Holdout set
Nested cross-validation
Data leakage
Model selection
grouped split
time-series split
preprocessing pipeline
out-of-fold prediction

Learn next

Draw a small fold assignment and mark where every transformation learns its parameters. Then redesign the split for repeated users and time-ordered records. Study nested cross-validation and metric aggregation next, so assessment of a chosen model is separated from the process that selected it.

Nested cross-validation

Sources

Scikit-learn, cross-validation guide: https://scikit-learn.org/stable/modules/cross_validation.html Scikit-learn, common pitfalls and data leakage: https://scikit-learn.org/stable/common_pitfalls.html Bates, Hastie and Tibshirani, Cross-validation: what does it estimate and how well does it do it?: https://arxiv.org/abs/2104.00673

Previous: Cross-sectional regression Next: Crossover point