What is the difference between validation and test sets?

Both are held out of training; the validation set guides tuning during development, the test set is touched once for the final honest score.

How big should the holdout be?

Big enough that the score's uncertainty is acceptable, commonly 20 to 30 per cent, with rare-outcome problems needing more.

What is data leakage?

Any path by which information from the holdout shapes the model, from direct tuning to subtle preprocessing choices, and it inflates every reported score.

AI Glossary / AI and machine learning / Beginner

Holdout data

Definition

Holdout data is the portion of available data deliberately kept away from training, reserved to test the finished model on cases it has never seen, so the score means something.

What it means

Training a model on data and then grading it on the same data is marking homework with the answer key open. Holdout data is the sealed exam: a slice of the data nobody touches until judging time.

The split is usually random and one-time: perhaps 70 or 80 per cent for training, the rest held out. The discipline matters more than the ratio. Anything that shapes the model, feature choices, settings, architecture, may only see the training portion.

The deepest discipline is resisting the peek. Every time someone checks the holdout and adjusts the model in response, the holdout leaks into development and its verdict weakens. A much-consulted holdout quietly becomes training data with a better reputation.

Many projects use two reserved slices: a validation set consulted during development for tuning, and a test set touched once at the end for the honest number. Holdout is the umbrella term for data kept out of training; the roles differ in how often they may be consulted.

For a business, holdout data is the difference between a claim and a measurement. Any accuracy figure worth acting on carries an implicit footnote: scored on data the model never trained on, by people who kept it sealed.

How it works

Decide the split before modeling begins, and document it: which records are training, which are held out, by what rule and at what date.

Keep the holdout physically or procedurally separate: separate tables, separate access, or at minimum a flag nobody joins against in development.

Tune and iterate against the training data and a development validation slice, never against the final holdout.

Score the finished model on the holdout once, report that number, and treat any later change as a new model needing fresh holdout discipline.

Refresh holdouts over time: an old holdout tells you about the old world, and drift demands new sealed exams.

Three practical examples

A bank that trains on 24 months of applications and seals the most recent 3 months as holdout, scored once before launch.
A retailer whose data team keeps the test slice in a separate schema that only the evaluation pipeline may read.
A vendor evaluation where the buyer supplies its own holdout and the vendor's reported accuracy is measured on it, not quoted from a brochure.

Think of it this way

Think of a wine competition where the judges' glasses are poured in another room. The winemakers shape the wine without seeing the judging table, and the verdict means something because of that wall. Holdout data is the other room.

The limit: the wall holds only while nobody drills through it. Each peek adjusts the wine to the judges, and enough peeks turn the competition into a rehearsal, with the same names and a worse meaning.

Worked example

the one-time exam

Suppose a telecom tunes a churn model through forty iterations. The figures are illustrative.

Each iteration is judged on a development slice, and only the final candidate touches the sealed test month: 88 per cent accuracy, reported to the board.

A colleague proposes one more tweak based on the test result, and the data lead refuses: one tweak turns the exam into a practice paper.

The refusal looks pedantic until a rival project, which tweaked freely, sees its 95 per cent collapse to 81 per cent live, while the sealed-exam model holds at 87 per cent.

Illustrative case study

A fictional startup showed investors a 94 per cent accuracy figure, achieved after months of enthusiastically consulting the test set, a habit the team did not recognize as leakage because nobody had written the rule down.

The pilot customer measured 71 per cent on its own data, and the fundraising conversation changed topic permanently.

The invented lesson: a holdout is a procedure, not a folder. If the team can see it, touch it and tune against it, you have training data with a ceremonial name.

Common misconceptions and mistakes

A holdout set is optional for small projects. It is the only honest evidence a project of any size has; small projects need it more, because their margins for error are thinner.
Random splitting is always right. Time-ordered problems need time-ordered holdouts, since random splits let the model peek at the future through leaked patterns.
One holdout lasts forever. The world drifts; yesterday's sealed exam measures yesterday's skill, and live models need fresh holdouts on fresh data.

Business use

Write the holdout rule into the project charter: what is sealed, who may touch it, and what touching it costs.

Ask for the holdout story behind every accuracy claim: how much, how chosen, how many times consulted.

Score final candidates on recent time-window holdouts for any problem where the world moves.

Check your understanding

Why does consulting the holdout during development destroy its value?

Answer: Because each adjustment made in response to the holdout score trains on it indirectly, so the sealed exam gradually becomes rehearsed material and its verdict stops predicting live performance.

Related terms

Learn next

Be Productive Using AI builds the habit of asking how any impressive AI number was measured before relying on it: https://aiforbusinessandlife.com/academy/be-productive-using-ai. [C1]

Sources

[S2] Google, Machine Learning Glossary. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Holdout cross-validation Next: Holdout set