Can validation examples ever be used in final training?

Yes, after selecting a procedure, some workflows refit using the combined development data. Final evaluation must still respect the planned separation.

Is a validation set required for every algorithm?

A separate fixed set is not universal; cross-validation and other designs can serve the assessment role. The need for credible evidence remains.

Should validation examples be easier than training examples?

They should represent the assessment objective. Deliberately choosing easy cases creates a misleading picture of the intended operating conditions.

AI Glossary / Causal inference and experimentation / Beginner

Validation set

Definition

A validation set is a collection of examples used to assess candidate models during development, separately from the examples used to fit each candidate. Its results guide choices such as hyperparameters, features, thresholds and training checkpoints. It is sometimes called a development set.

What it means

Training data helps fit a candidate's parameters. Validation data helps decide which candidate or procedure to choose. Test data supports a subsequent assessment of the selected procedure. These are roles in an evaluation design, not inherent properties of the records.

A validation set influences the final result even when its examples never appear in a gradient update. Choosing a configuration because it scored well on those examples uses their information.

The set must represent the question being asked. Assessing performance on new customers differs from assessing new records for existing customers. Predicting next month's demand differs from reconstructing a randomly omitted historical observation. The splitting rule should reflect those differences.

How it works

Define the prediction time, target population and unit that must remain separate. Then allocate examples using an appropriate rule. Random or stratified splits can suit some independent-example tasks. Grouped splits keep related records together; chronological splits respect the order of available information.

Fit learned transformations, such as scaling or feature selection, on the training portion. Apply the resulting transformation to validation examples. Fitting those steps on all records before splitting can allow validation information to influence the candidate being assessed.

Use the validation results to compare candidates under a consistent protocol. Record major decisions and avoid repeatedly redesigning the measure to favor the latest result. Extensive adaptation can reduce the set's value as evidence about unseen data.

Cross-validation rotates the assessment role across folds instead of relying on a single permanent validation portion. Within each fold, the fitting and assessment roles still need separation. Once a procedure is selected, refitting on a larger development dataset may be appropriate, while preserving the intended independence of a final test.

Three practical examples

A parts classifier keeps photographs of the same physical component in one split to avoid near-duplicate views crossing the boundary.
A forecasting team uses an earlier period for fitting and a later period for validation, using only information available at each forecast origin.
A support classifier holds out whole organizations when the intended launch serves organizations not represented in training.

Think of it this way

Think of keeping practice questions aside to choose a study method. If you repeatedly change your study plan to improve on those questions, they have become part of your preparation process.

The analogy does not imply that one universal fraction should be set aside. The amount and type of validation evidence depend on the task, the variation in examples and how the results will be used.

Worked example

Rows are not always independent cases

An invented image collection contains 100 physical parts, each photographed ten times. There are 1,000 image rows. A random row split may place pictures of the same part in both training and validation, making the assessment partly about recognizing already-seen objects.

Suppose the intended task concerns previously unseen parts. A group split assigns 70 parts to fitting, 15 to validation and 15 to a final test. The resulting image counts are 700, 150 and 150, assuming each part still has ten images.

The important change is the boundary around the physical object, not the convenient percentages. The validation sample contains only 15 held-out parts, so its 150 images should not be treated as 150 fully independent objects when interpreting uncertainty.

Illustrative case study

A fictional planning service, Maple Routes, builds arrival-time predictions from a year's delivery records. Its validation set is a random sample from the whole year, while training includes later deliveries than some validation trips.

The intended use is forecasting future trips as information arrives. The team creates a chronological development exercise and checks which features would actually have been available before each departure. It removes a field populated after delivery completion.

The revised validation set produces less flattering but more relevant evidence. The team also retains difficult routes instead of silently excluding them. The case illustrates why the data boundary and information timing matter as much as the model settings.

Common misconceptions and mistakes

A validation set is not simply any unused spreadsheet. Its labels, coverage and relationship to the training data determine what conclusions it supports.

Stratifying class proportions does not automatically handle duplicated objects, repeated customers or time dependence. These require their own separation decisions.

There is no universal correct split percentage. More validation examples can improve measurement while leaving fewer examples for fitting, and rare conditions may require targeted assessment. A large row count does not compensate for a small number of independent groups or missing operating conditions.

Business use

Document the validation set as a reusable evaluation asset: its purpose, sampling process, exclusions, labels, groups, period and known limitations. Track which development decisions have used it.

Set aside evidence for important scenarios that an overall average could hide. Revisit representativeness when the service expands or inputs change. When a team needs stronger final evidence after extensive tuning, plan fresh or appropriately protected assessment data instead of relabeling the familiar validation set as a test.

Check your understanding

Twenty pictures of each product are split randomly by image, but the service must classify entirely new products. What boundary should the team reconsider?

Answer: The product identity. Keeping all pictures of a product within one split provides a more relevant assessment of performance on unseen products.

Related terms

Training set
Test set
Development set
Cross-validation
Data leakage
group split
temporal validation
hyperparameter tuning
representative sample

Learn next

Draw the relationships between records in your dataset: shared objects, customers, sessions and times. Decide which relationships must not cross the assessment boundary. Then document the fitting, selection and final-test roles before experimenting with model configurations.

Sources

Sources: Google Machine Learning Glossary, validation set: https://developers.google.com/machine-learning/glossary#validation-set Aalto Dictionary of Machine Learning, validation set: https://aaltodictionaryofml.github.io/ADictML_Main.pdf Scikit-learn, cross-validation, groups and time series: https://scikit-learn.org/stable/modules/cross_validation.html Scikit-learn, data leakage and preprocessing: https://scikit-learn.org/stable/common_pitfalls.html

Previous: Validation loss Next: Validity