Is there a mandatory test-set percentage?

No. Size depends on available data, variability, rare cases and the precision required for evaluation.

Is validation data the same as test data?

Their roles differ: validation guides development, while a final test assesses the selected approach.

Can a model be trained on all data after testing?

Sometimes a final production fit uses more data, but then the exact deployed model differs from the tested fit and the evaluation implications should be clear.

AI Glossary / AI and machine learning / Beginner

Test set

Definition

A test set is a collection of examples reserved for assessing a model after its development choices have been made. It helps estimate how the model performs on cases that did not guide training or model selection.

What it means

A model can perform well on examples it has already seen without learning a useful general pattern. The test set creates a separate examination of its behavior.

Its independence is about the whole development process, not only the final training command. If a team repeatedly changes features after seeing test results, information from the test set has influenced the model indirectly.

A test set should also represent the intended use. A collection of easy cases from an old product range may be separate from training data yet still tell little about next month's customers.

For business readers, a reported test score is therefore a starting point for questions. Ask where the cases came from, what decisions were made before they were examined and which kinds of errors the score hides.

How it works

The team defines the population and decision being evaluated, then chooses a separation strategy. Independent records may permit a random split. Repeated customers, related documents or chronological prediction may require grouping or time-based separation.

Training data fits the model. Validation data or cross-validation guides choices such as settings, features and thresholds. The test set is used for a final assessment under a documented procedure.

Preprocessing must respect the split. For example, learned scaling statistics or feature selection should not use the test records to make the training problem easier.

The evaluation reports appropriate metrics, uncertainty and relevant subgroup results. If the team responds to a disappointing test by redesigning the model, that test has become part of development history; another independent assessment may be needed for a fresh final claim.

Three practical examples

Message routing: hold out complete later conversations so near-duplicate messages from one exchange do not appear on both sides of the evaluation.
Equipment prediction: reserve machines or operating periods that reflect where the model will next be used.
Document extraction: test on representative unseen document layouts and report field-level errors as well as overall completion rates.

Think of it this way

Think of a final examination whose questions were not used as practice exercises. It can reveal whether learning extends beyond memorized answers.

The analogy has a limit. An exam drawn from the wrong syllabus can be independent yet irrelevant. Similarly, a test set must reflect the operational task, including difficult cases and realistic input quality. Independence alone does not guarantee representativeness.

Worked example

Reading beyond overall accuracy

Suppose an invented test set contains 1,000 messages: 950 routine and 50 urgent. A model that labels every message routine is correct on 950 cases, giving 95% accuracy.

It nevertheless misses all 50 urgent messages. Its recall for urgent messages is zero. If the business purpose is to identify urgency, the headline accuracy conceals total failure on the main objective.

The team should report how many urgent cases are detected and how many routine cases are incorrectly escalated. It should also examine whether 50 urgent examples provide enough evidence for the precision required in its performance claims.

The figures illustrate why a test set needs both suitable composition and suitable metrics. They are not a benchmark or a recommended class balance.

Illustrative case study

A fictional software team checks the same test collection after every model revision. Over time, it adds rules for unusual phrases that caused errors in that collection. The reported score steadily improves.

When a fresh collection arrives, the gains are much smaller. The team recognizes that the original test set became a development resource. It keeps the error analysis as useful learning, but establishes a new evaluation process with a protected final assessment.

The lesson is that repeated attention can reduce the independence of a test without any record being explicitly inserted into the training file.

Common misconceptions and mistakes

Any unused row is a valid test example. Related records, duplicates or future information may still create leakage.
A high test score guarantees future performance. Population changes and operational differences can weaken the result.
The test set should guide every improvement. Development data serves that role; repeated test-driven changes compromise a final independent claim.

Business use

A test set supports evidence-based release decisions and comparisons between suppliers or model versions. Its value comes from a credible evaluation design, not simply from having a file named test.

Document the split, date range, excluded cases and metrics. Include the cost of consequential errors and the workflow for uncertain predictions. Retain enough detail to reproduce the assessment, while continuing to monitor live outcomes after release.

Check your understanding

A team chooses a threshold by trying many values on the final test set. Is that set still untouched evidence for the chosen threshold?

Answer: No. The threshold was selected using those outcomes. A separate evaluation is needed for an independent assessment of the selected setting.

Related terms

Learn next

Read validation set and data leakage. Ask for a diagram showing which records influenced training, model selection and final reporting, including preprocessing and threshold choices.

Sources

scikit-learn, Cross-validation: evaluating estimator performance: https://scikit-learn.org/stable/modules/cross_validation.html. The separation of training, selection and final evaluation was consulted. Examples and arithmetic are original.

Previous: Test loss Next: Test statistic