Must every project use three fixed datasets?

No. Cross-validation and other designs can provide development feedback, but the distinction between selecting a procedure and independently evaluating it still matters.

Can training and validation data later be combined?

Sometimes, after choices are fixed. The final refitting procedure must be specified and evaluated without using test feedback to tune it.

Does a good test score guarantee deployment performance?

No. It supports performance under the test conditions; new environments, changed behavior, or missing cases can produce different results.

AI Glossary / Computer vision and multimodal AI / Beginner

Train/validation/test split

Definition

A train/validation/test split divides a dataset into parts with different roles: training fits a model, validation guides development choices, and testing estimates performance after those choices are fixed. The separation helps distinguish learning useful patterns from adapting to the particular examples used to judge success.

What it means

Training data is the material a model learns from. Validation data provides feedback when choosing settings, comparing designs, selecting a checkpoint, or adjusting a decision threshold. Test data is reserved for a final evaluation of the resulting procedure.

These names describe how information is used, not merely the names of three folders. If a team repeatedly checks test results and changes its model to improve them, that test set has become part of development. Its score is no longer an independent assessment of those choices.

The split should reflect the intended use. Recognizing new photographs of familiar products is different from recognizing entirely new product families. Both are reasonable questions, but they require different separation rules and support different claims.

How it works

First define the unit that must remain independent: perhaps a photograph, physical item, person, recording session, location, or time period. Assign related examples together when separating them would let information about an evaluation case enter training.

Fit learned preprocessing using training data. Apply the resulting transformations to validation and test examples without learning from those held-out sets. During development, use validation results to select the complete procedure, including thresholds and stopping rules.

Then evaluate the frozen procedure on the test set and report relevant metrics, sample counts, and limitations. Cross-validation can rotate training and validation roles within development data, while an untouched final test set remains separate. The method and proportions depend on the amount, dependence, and diversity of available evidence.

Three practical examples

1. A defect detector receives several photographs of each manufactured component. Splitting by component keeps views of the same scratch together, so the evaluation asks whether detection transfers to other components.

2. A wildlife classifier uses images from camera traps. Holding out whole locations tests transfer to unfamiliar backgrounds; randomly mixing images from every location answers an easier and different question.

3. A document reader will process next quarter's incoming forms. A chronological evaluation can reveal problems caused by changing layouts and suppliers that a random historical split might conceal.

Think of it this way

Think of preparing for an examination: study materials support learning, practice questions guide improvement, and a separately prepared examination checks performance. Rehearsing the examination itself changes what its score means.

The analogy has limits. Dataset examples may share hidden dependencies, and a future deployment may differ from any examination assembled today. Separation improves the evidence; it does not guarantee that the evaluation covers every future situation.

Worked example

Suppose an image dataset contains 1,000 physical components, each photographed from five angles: 5,000 images altogether. The project must recognize defects on new components of the same general kinds.

An illustrative split assigns 700 components to training, 150 to validation, and 150 to testing. Because all five views stay together, the corresponding image counts are 3,500, 750, and 750. No component appears in more than one part.

Two candidate models correctly classify 690 and 705 validation images. Their image-level accuracies are 92% and 94%. The team selects the second using its predefined evaluation rule, freezes the procedure, and obtains 675 correct test predictions out of 750: 90% test accuracy.

The validation advantage does not promise 94% future accuracy. Moreover, those 750 test images represent only 150 independent components under this grouping assumption. Uncertainty estimates and business reporting should respect the dependence between views. A component-level decision rule may also matter more than individual-image accuracy.

Illustrative case study

Fictional case: RivetScope develops a visual inspection model using photographs collected during factory visits. An initial random image split looks encouraging, but an audit finds near-identical views of individual parts in different partitions.

The team rebuilds the split around physical part identifiers and reserves a later visit for an additional deployment check. It also records lighting conditions and checks whether errors cluster under one camera setup.

The revised evaluation gives the team a clearer basis for deciding what to collect next. This hypothetical case illustrates a change in evidence quality, not a measured performance improvement or a claim that every project needs the same partition design.

Common misconceptions and mistakes

An 80/10/10 split is not a universal standard. A tiny test set may be too unstable for a decision, while a large but unrepresentative one can be confidently misleading.

Randomness does not remove leakage automatically. Duplicate files, adjacent video frames, and multiple observations of one subject can cross a random boundary. Stratification can balance class proportions, but it does not by itself keep related groups apart.

Another mistake is treating every held-out result as permission to keep tuning. After substantial test-informed changes, a fresh independent evaluation may be needed to assess the revised procedure credibly.

Business use

For a business, the split defines the strength and scope of a model's performance claim. Record which entities were separated, when the test data was collected, and which decisions used validation feedback.

Agree on the intended deployment question before comparing scores. A supplier promising accuracy on unseen sites should provide evidence that actually holds out sites. Keep the split specification and dataset version with the evaluation report so later comparisons remain interpretable.

Check your understanding

A team chooses its best model by comparing ten models on the supposed test set. What role has that set actually played?

Answer: It has supplied model-selection feedback, so it has functioned as validation data for that choice. An independent final evaluation is still needed for a credible assessment of the selected procedure.

Related terms

Learn next

Identify the real-world entity behind each row in a small dataset. Design two splits that answer different deployment questions, and explain which duplicates, groups, and time boundaries each must keep separate.

Sources

Primary references: scikit-learn, Cross-validation: evaluating estimator performance, https://scikit-learn.org/stable/modules/cross_validation.html ; Common pitfalls and recommended practices, https://scikit-learn.org/stable/common_pitfalls.html . These documentation pages support the separation and leakage principles. The component counts, fictional organization, and teaching examples are original illustrations.

Previous: Trail Next: Trained model