Do all outer folds choose the same settings?

No. Different training portions may favor different candidates, which can reveal selection instability.

Is nested cross-validation always necessary?

No. A properly separated validation and test design may be appropriate. The choice depends on data, selection complexity, and evaluation goals.

Does it train the final deployment model?

It trains evaluation models. A final selection and fit on the intended development data is a separate step.

AI Glossary / Causal inference and experimentation / Intermediate

Nested cross-validation

Definition

Nested cross-validation is an evaluation procedure that places a model-selection process inside an outer cross-validation process. The inner loop chooses settings using only the current outer training data. The outer loop evaluates the resulting choice on data withheld from that entire selection process. Its purpose is to assess the performance of a training-and-selection procedure while reducing the optimistic bias caused by choosing and reporting the best result on the same validation data.

What it means

Trying many model settings creates a selection problem. Some settings can look especially good on a particular validation sample partly by chance. Reporting that winning validation score as if it were an independent assessment can overstate expected performance.

Nested cross-validation separates two questions: which option should be selected using the available training data, and how well does that selection procedure work on held-out examples? Each outer split repeats both steps.

The result describes the procedure, including its search space and preprocessing, under the chosen data-splitting assumptions. It is not a guarantee about future environments, nor does it eliminate uncertainty from small or unrepresentative datasets.

How it works

Divide the dataset into outer folds. For one outer iteration, reserve an outer test fold and give the remaining examples to an inner cross-validation search. The inner search compares candidate hyperparameters or model choices without seeing the outer test fold.

Choose the candidate using the inner results, then refit it on the entire outer training portion. Evaluate that fitted model once on the outer test fold. Repeat for the other outer folds and summarize the outer scores.

Any learned preprocessing, feature selection, or other data-dependent choice belongs inside the appropriate training process. Otherwise information can leak across boundaries. After estimating performance, a team can run its selection procedure on all development data and fit a final model. That final fit is separate from the outer evaluation fits and may select a different setting.

Three practical examples

1. Small classification study: researchers compare several regularization strengths while using outer folds to assess the complete selection procedure.

2. Feature selection: a business tests how a pipeline chooses useful customer features, ensuring feature selection is repeated within training folds rather than performed once on the full dataset.

3. Model-family comparison: engineers include two candidate algorithm families in an inner search, then evaluate the selected family and settings on each outer test fold.

Think of it this way

Think of a competition in which practice rounds determine a team's lineup, and a separate match tests that lineup. The match result is more informative than reporting the best practice score. The analogy is limited because nested cross-validation reuses observations across different training and testing roles in separate iterations; its scores are not results from wholly independent competitions.

Worked example

Suppose a fictional dataset has 120 independent examples. A team uses three outer folds, two inner folds, and four candidate settings. Each outer iteration holds out 40 examples and gives 80 to the inner search.

Within that 80-example portion, each inner split trains on 40 examples and validates on the other 40. Four candidates across two inner folds require eight candidate fits. The winning setting is then refitted on all 80 outer training examples, adding one fit. Across three outer iterations, this requires 3 × (8 + 1) = 27 fits, excluding any final production-model search and fit.

Suppose the selected models correctly classify 32, 34, and 30 of their respective 40 held-out examples. Their outer accuracies are 80%, 85%, and 75%, averaging 80%. This summarizes the evaluated selection procedure. The best inner score from one iteration should not replace those outer results. The toy example assumes independent observations; grouped or time-ordered data require suitable splitting rules in both loops.

Illustrative case study

Fictional case: Larch Subscriptions wants to predict which service requests need specialist handling. Its analysts try several feature sets and tuning options. An initial report highlights the highest cross-validation score used to choose among them.

A reviewer asks for evaluation that includes the effects of selection. The analysts place the full search inside an outer loop and group requests by customer, keeping related requests together at each relevant boundary. They report the outer results and explain how much training data each outer model saw. This case illustrates a more defensible assessment process, not a real organization's performance improvement.

Common misconceptions and mistakes

Two cross-validation runs are not automatically nested. The inner process must occur separately within each outer training partition. Similarly, selecting features once before both loops can undermine the intended separation.

Do not repeatedly adjust the search procedure after inspecting outer results and continue treating those results as untouched evidence. Such choices can adapt the workflow to the outer data too. Record exploratory changes and obtain fresh evaluation when necessary.

Another mistake is assuming nesting fixes a poor split. Random folds remain inappropriate when they leak future information or closely related examples. Computational cost also matters: a large search inside many folds can require substantial resources.

Business use

Nested cross-validation is useful when model selection is extensive and an honest estimate of that selection process matters. It helps distinguish a promising development score from evidence suitable for comparing approaches.

Business reports should describe the outer and inner split strategies, candidate search, metric, and any excluded data. Explain that the eventual model may train on more examples than each outer model. The value is a clearer estimate with documented assumptions, not an automatic promise that nesting produces a particular score or superior model.

Check your understanding

An analyst tunes on all examples before running an outer evaluation of the chosen settings. Is that a properly nested assessment of tuning?

Answer: No. The outer test examples already influenced the choice. Repeat tuning within each outer training partition to evaluate the selection procedure as intended.

Related terms

Cross-validation
Model selection
Data leakage
Pipeline
Test set
Selection bias
hyperparameter tuning
group cross-validation

Learn next

Draw a three-outer-fold, two-inner-fold diagram. Mark which observations each fitting, selection, and scoring step can access. Add a learned scaling step and place it where it cannot inspect held-out observations.

Sources

Scikit-learn, Nested versus non-nested cross-validation: https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html

Scikit-learn, Cross-validation: evaluating estimator performance: https://scikit-learn.org/stable/modules/cross_validation.html

The primary documentation supports the loop separation and splitting cautions. Fit counts, results, fictional case, and exercise are original illustrations.

Previous: Nested case - control study Next: Nested hypothesis