What is the difference between leakage and a strong predictor?

A strong predictor was knowable at prediction time; a leaked field encodes information created by or after the outcome.

How does leakage survive a holdout?

The leaked field exists in both training and holdout data, so the exam shares the answer key with the lessons unless the split respects time.

What is the best defense?

A production-time audit of every field, plus a time-ordered final test that mimics true deployment.

AI Glossary / AI and machine learning / Intermediate

Label leakage

Definition

Label leakage is the accidental inclusion, in a model's inputs, of information that is derived from or equivalent to the answer, so the model learns to read the answer key and collapses when it is absent in the real world.

What it means

Leakage is the quiet way models cheat. Somewhere in the feature list sits a field that could only be known after the outcome: a collections note in a default model, a refund flag in a satisfaction model, the discharge disposition in a readmission predictor.

The model is innocent and the results are spectacular. Training and holdout scores soar, because the leaked field effectively hands the model the answer, and everyone celebrates until deployment, where the answer-derived field does not yet exist and the model performs like an ordinary mortal or worse.

Leakage survives good intentions because it hides in derived and operational fields: timestamps from after the decision, aggregates computed over the full period, codes written by processes that only run once the outcome is known.

The defense is temporal discipline: for every field, ask when it became knowable, and allow nothing into training that was not knowable at prediction time. The question is simple, and answering it honestly is most of the battle.

For a business, label leakage is the first explanation to check for any suspiciously excellent result: a score far above what the problem's history suggests has a leaked answer key until proven otherwise.

How it works

List every candidate field's production time: when in the real process does this value first exist?

Exclude or rebuild anything born after prediction time, including aggregates computed over periods that extend past the decision.

Inspect top-importance fields with suspicion: a single field carrying the model is a classic leak signature.

Test on genuinely future data: time-ordered holdouts catch leaks that random splits let through, because the leaked field correlates across the whole era.

Re-audit after every pipeline change: new joins and derived fields are how leaks re-enter a cleaned model.

Three practical examples

A default model whose top field, account restructure code, exists only because the collections process began, a perfect leak removed before launch.
A churn model accidentally trained on twelve-month revenue aggregates that included the months after the cancellation.
A readmission predictor leaking the discharge ward, a value recorded only once the outcome the model predicts had already been decided.

Think of it this way

Think of a student who smuggles the mark scheme into the exam inside a formula sheet. The results are excellent, the teacher is delighted, and the student knows nothing, which is discovered expensively at university. Leakage is the formula sheet.

The limit: the sheet is rarely labeled mark scheme. It arrives as an innocent operational field, and only the question when was this first knowable separates the smuggled answers from the honest working.

Worked example

the 97 per cent model

Suppose a bank's new default model validates at 97 per cent, twelve points above anything the problem has ever sustained. The figures are illustrative.

A leak hunt finds the top field: days since last collections activity, populated only for accounts already in trouble.

Removing it drops validated performance to 86 per cent, a disappointment and a rescue, because 86 per cent is the truth the model will actually live on.

The review board adopts a new custom: results this good trigger a leak hunt before they trigger a celebration.

Illustrative case study

A fictional hospital trained a readmission model that achieved startling accuracy, deployed it, and watched it add no value at all: the decisive field, discharge medication review completed, described a step taken after the decision to discharge.

The model had learned to recognize patients who were already safely out the door, a talent with no use at the moment of decision.

The invented lesson: a field can be perfectly recorded, perfectly clean and perfectly useless, because it exists on the wrong side of the decision. Timing is data quality.

Common misconceptions and mistakes

Leakage only happens through obvious fields. It hides in derived features, aggregates, timestamps and operational codes more often than in anything named outcome.
High accuracy proves there is no leak. Spectacular accuracy is the leak's calling card; suspicion should scale with celebration.
Random holdouts catch leakage. They often do not, because the leak pattern spans the era; time-ordered testing is the honest sieve.

Business use

Make when was this field first knowable a standard sign-off question for every model.

Treat sudden performance jumps, in your models or a vendor's, as leak investigations before success stories.

Use time-ordered holdouts for any model whose fields have operational histories.

Check your understanding

Why does label leakage produce models that test brilliantly and fail in deployment?

Answer: Because the leaked field carries the answer into the exam but does not exist at prediction time in the real world, so the tested performance measures answer-copying, not prediction.

Related terms

Learn next

AI for Business Owners covers how to challenge a suspiciously good model result before it reaches a customer: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S2] Google, Machine Learning Glossary. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Label bias problem Next: Label noise