How many models are fitted for n observations?

The straightforward leave-one-out procedure fits n models, each using n − 1 observations.

Is the final deployed model trained on only n − 1 records?

Not necessarily. After selecting the procedure, a final model can be fitted on all available development records.

Is leaving out one customer the same as leaving out one row?

Only if each customer contributes one row. With repeated customer records, holding out a customer is a grouped evaluation.

AI Glossary / Causal inference and experimentation / Intermediate

Leave-one-out cross-validation

Definition

Leave-one-out cross-validation evaluates a learning procedure by repeatedly training on all observations except one and testing on the single omitted observation. With n observations, it performs n separate fits so every observation is held out once. It is a special case of k-fold cross-validation with k equal to the dataset size. The procedure uses almost all available data in each fit, but that advantage must be weighed against computation and the data's dependence structure.

What it means

When a dataset is small, setting aside a large permanent test portion can leave little data for learning. Leave-one-out offers an alternative: every record contributes one prediction made by a model that did not train on that record, while each fit uses the remaining records.

This does not mean the procedure reveals an exact future error rate. The fitted models share most of their training observations, and the held-out records may not represent deployment conditions. Leaving out one row is particularly misleading if nearly identical or related rows remain in training. The unit omitted must match the generalization question being asked.

How it works

For each observation, remove it from the training partition. Fit all learned preprocessing and the model using the other observations, predict the omitted target, and record the relevant loss or prediction. Repeat from an appropriately fresh fit for every record.

Aggregate the held-out results using a metric that is defined for the intended calculation. Single-observation folds cannot support every familiar metric individually; for example, a per-fold comparison requiring both classes is impossible when a fold contains one case.

If the results guide model selection, account for that selection when assessing final performance. As with other cross-validation methods, a pipeline must keep information from the held-out item out of training, including feature selection and preprocessing parameters.

Three practical examples

1. Small regression exercise: researchers estimate prediction error by fitting on all but one experimental observation at a time, then inspect which cases are difficult to predict.

2. Algorithm comparison: students compare simple learning procedures on the same omitted-record sequence, recording both predictive losses and the cost of repeated fitting.

3. Sensitivity investigation: an analyst examines whether predictions for certain observations are consistently poor when those observations are excluded from learning, while avoiding claims of independent fold results.

Think of it this way

Think of preparing a separate rehearsal for every question, each time learning from all the other questions and answering the omitted one. The analogy is limited because a person would remember earlier rehearsals. A valid computational procedure must not carry knowledge from a previously held-out answer into the next fitted model. Nor does rehearsing many overlapping sets create a new independent dataset.

Worked example

Consider target values [2, 4, 5, 9] and a deliberately simple model that always predicts the mean of its training targets. Omitting 2 leaves a mean of (4 + 5 + 9)/3 = 6, giving absolute error 4.

Omitting 4 gives prediction 16/3, approximately 5.333, and error 1.333. Omitting 5 gives prediction 15/3 = 5 and error 0. Omitting 9 gives prediction 11/3, approximately 3.667, and error 5.333.

The leave-one-out mean absolute error is approximately (4 + 1.333 + 0 + 5.333)/4 = 2.667. A final mean predictor fitted to all four observations would predict 5. The evaluation fits and the final model therefore have different training sets and need not make identical predictions.

Illustrative case study

Fictional case: Rowan Lab has a small collection of repeated measurements from several devices. It initially uses leave-one-out by row and obtains reassuring predictions because each omitted reading resembles other readings from the same device still in training.

The intended application, however, concerns a new device. The team changes the evaluation unit to devices, holding out all readings from one device together. That is leave-one-group-out, not ordinary leave-one-observation-out. It also compares the computation with a suitable grouped k-fold design. The case shows that maximizing training-set size is less important than representing the actual deployment question correctly.

Common misconceptions and mistakes

Do not assume leave-one-out is always more accurate as an error estimator than a smaller number of folds. Its many highly overlapping fits can produce a variable estimate, and performance depends on the learning procedure and dataset.

Avoid fitting transformations on the complete dataset before the loop. The omitted observation must remain excluded from every learned preparation step.

Do not average undefined single-record metrics or interpret n fold scores as independent experiments. Also distinguish leaving out one record from leaving out one person, device, location, or time block. Those choices evaluate different kinds of generalization.

Business use

Leave-one-out can be useful for small datasets and inexpensive models when observations are suitable for individual omission. Its appeal decreases when training is costly or records are strongly related. A dataset with 10,000 observations requires 10,000 fits in the straightforward procedure, before considering model tuning.

Business teams should compare the evaluation cost with alternatives and ask whether the resulting estimate addresses the intended use. Document preprocessing, the omitted unit, the aggregation metric, and any model-selection steps. The output is evidence about a learning process under a particular sampling design, not a guarantee for a final deployed system.

Check your understanding

A dataset contains 30 observations. In ordinary leave-one-out cross-validation, how large is each training set and each evaluation fold?

Answer: Each fit trains on 29 observations and evaluates on one. There are 30 such fits, and every observation supplies one held-out prediction.

Related terms

Learn next

Reproduce the four-record mean-predictor calculation, then compare it with the model fitted on all records. Identify whether your real dataset contains repeated entities or temporal dependence. If it does, specify an omission unit that reflects the application before choosing leave-one-out as the evaluation design.

Sources

Primary implementation source: scikit-learn, Cross-validation, Leave One Out, https://scikit-learn.org/stable/modules/cross_validation.html#leave-one-out-loo . The fitting procedure, computational comparison, and evaluation caveats were consulted. The constant-mean worked example and fictional device case are original illustrations. This entry concerns ordinary observation-level leave-one-out and explicitly distinguishes grouped alternatives.

Previous: Least-to-most prompting Next: Lebesgue integral