Does k mean the number of training examples?

No. It is the number of folds and, in a single k-fold run, the number of separate evaluation fits.

Can I keep the model with the best fold score?

That is not the usual purpose. The folds assess a learning procedure; a final model is generally fitted after the procedure is selected.

Is ordinary random k-fold suitable for every dataset?

No. Time dependence, related records, and class structure can require different splitting strategies.

AI Glossary / Causal inference and experimentation / Beginner

k-fold cross validation

Definition

K-fold cross validation is an evaluation procedure that divides data into k non-overlapping parts, or folds, and repeatedly trains a model on all but one fold while evaluating it on the remaining fold. Each fold serves as the held-out part once. The resulting scores help estimate how a learning procedure performs on unseen examples drawn under comparable conditions. Appropriate splitting and prevention of data leakage are essential to making that estimate useful.

What it means

A single train - test split can give a result that depends heavily on which examples happened to be held out. K-fold cross validation rotates that role through the available data, providing several evaluations rather than relying on one partition.

It evaluates repeated fits of a learning procedure, not one model that gradually sees all the data. Each fit must start appropriately afresh, using only its training folds for learning. Cross validation does not manufacture more independent observations, and it does not guarantee that future data will resemble the dataset. Its usefulness depends on whether the split reflects the situation the model will actually face.

How it works

Choose k and construct folds suitable for the data. For ordinary independent examples, the parts are roughly equal in size. For each fold, fit preprocessing and the model using the other k − 1 folds, then measure predictions on the held-out part. Summarize the scores and inspect their variation.

Classification may use stratified folds to preserve class proportions. Repeated records from the same person, device, or organization may require group-aware splitting. Forecasting needs a time-aware design that respects what information was available when a prediction would be made.

If cross-validation results guide model selection, an independent final test or a suitable nested procedure can help assess the selected process. After selection, a final model can be fitted on the available development data.

Three practical examples

1. Ticket routing: a team compares candidate classifiers using the same stratified folds and a metric that reflects its class distribution.

2. Equipment prediction: engineers keep records from each machine together when estimating performance on machines absent from training, adapting the splitting strategy to the intended use.

3. Small regression study: researchers compare preprocessing and model choices with repeated held-out folds, fitting every transformation separately within each training partition.

Think of it this way

Think of rotating which group of practice questions is kept aside while preparing independently with the others. Each group gets a turn as the unfamiliar test. The analogy is limited: a human would remember earlier questions, whereas the model fits must avoid carrying learned information from earlier held-out folds. The folds also reuse much of the same training data, so their scores are not independent experiments.

Worked example

Suppose 100 independent records are divided into five folds of 20. Each fit uses 80 records for training and 20 for evaluation. Across the five evaluation folds, the numbers of correct predictions are 16, 18, 17, 15, and 19.

The fold accuracies are 80%, 90%, 85%, 75%, and 95%. Their mean is 85%. Because the folds have equal size, this also equals 85 correct held-out predictions divided by 100 records.

Each record was held out once and belonged to training sets four times, across separate fitted models. This does not mean the deployed model is one of those five fits or that 85% is guaranteed on new data. Unequal fold sizes or nonlinear metrics can make pooled and mean-fold summaries differ.

Illustrative case study

Fictional case: Elm Forecast builds a predictor from repeated readings across several machines. Randomly divided folds produce encouraging scores, but the intended deployment concerns newly installed machines. The original split lets readings from the same machine appear on both sides, making the evaluation easier than the deployment problem.

The team switches to groups defined by machine identity and rechecks its feature pipeline within each fit. It separately considers whether future-time prediction on known machines requires another evaluation design. The change makes the question explicit: generalizing to an unfamiliar machine differs from predicting another reading from a familiar one. No performance figure is claimed for this fictional project.

Common misconceptions and mistakes

Do not fit scaling, imputation, feature selection, or other learned preprocessing on the entire dataset before splitting. That can pass information from held-out data into training.

Avoid carrying the trained model from one fold into the next, which breaks the intended independence of each held-out evaluation from its training procedure.

Do not assume shuffling solves temporal or group dependence. Also avoid selecting many model variations using cross validation and then presenting the best score as an untouched final assessment. Selection itself uses information from the evaluation process.

Business use

Cross validation helps businesses compare modeling options when data are limited and a single split would be fragile. It also exposes variation across partitions that deserves investigation before a model is integrated.

Balance the number of fits against training cost, and use the same relevant splits for fair candidate comparisons. Document the unit being split, the metric, and whether the scores informed tuning. Ask whether the evaluation simulates future customers, future dates, or additional records from known entities. Those distinctions determine the credibility of the estimate more directly than choosing a familiar value of k.

Check your understanding

In five-fold cross validation, may an imputer learn replacement values from all five folds before evaluation?

Answer: No. For each fit it should learn those values only from that fit's four training folds, then apply them to the held-out fold. Otherwise the evaluation data influence preparation.

Related terms

Cross-validation
Validation set
Data leakage
Nested cross-validation
Model selection
stratified k-fold
group cross validation
time-series split

Learn next

Divide a small independent dataset into three folds and write out the training and evaluation membership for each fit. Add a learned preprocessing step and specify where it is fitted. Then identify whether the real application contains groups or time ordering that would require a different split.

Sources

Primary implementation source: scikit-learn, Cross-validation: evaluating estimator performance, https://scikit-learn.org/stable/modules/cross_validation.html . The k-fold mechanism, preprocessing guidance, and group/time distinctions were consulted. This entry describes the procedure rather than prescribing one split for all applications. The five-fold arithmetic and fictional machine example are original teaching illustrations.

Previous: K-edge-components Next: K-independent hashing