How do I spot leakage?

Unusually high scores, one feature dominating, or a sharp drop from test to live performance.

Is leakage the same as overfitting?

No; overfitting memorizes noise, while leakage uses information that will not exist at prediction time.

Can preparation steps leak?

Yes, scaling or filling gaps using the full dataset before splitting lets test data influence training.

Can leakage happen with text or images?

Yes; duplicated images or near-identical documents in both training and test sets give the same flattering effect.

AI Glossary / AI and machine learning / Intermediate

Data leakage

Definition

Data leakage is when information from outside a model's legitimate training view, often the answer itself, slips into its inputs, making test results look excellent and real-world performance disappointing.

What it means

A model should predict using only what will be known at the moment of prediction. Leakage breaks that rule by letting future or outcome information sneak into the inputs.

A classic case: predicting which customers will cancel, using a column recording whether the cancellation form was opened. The column is only filled after the decision, so the model looks brilliant and is useless.

Leakage also happens through careless preparation. Scaling or filling gaps using the whole dataset before splitting lets test data influence training, subtly flattering the results.

It is dangerous because it passes every normal check. The test score is high precisely because the test data leaked too.

For a business, leakage is the most common reason a promising model collapses in production. A suspiciously good result is more often a leak than a breakthrough.

There is a quieter form called target leakage through grouping: the same customer appearing in both training and test data lets the model recognize people instead of learning patterns.

How it works

Ask of every input: will this value be known at the moment of prediction, in live use, in the same form?

Split the data into training and test sets first, then do all preparation steps using only the training portion.

Respect time. For forecasts, train on the past and test on the future; random splits can leak the future into training.

Watch for proxies for the outcome: status fields, timestamps after the event, and identifiers that encode the result.

Be suspicious of results that look too good, and check which features dominate; a single near-perfect feature is a red flag.

Split by customer or by time rather than by row when the same entity appears many times, so no individual spans both sides of the split.

Three practical examples

Churn models using a cancellation-reason field that is only filled after customers leave.
Fraud models trained with a chargeback flag that arrives weeks after the transaction.
Demand models where tomorrow's promotion calendar leaks into features used to forecast today.

Think of it this way

Think of a student who finds the exam answers pinned inside the textbook. Their score is superb, and they have learned nothing. Put them in a real job and the gap appears at once.

The limit: leakage is rarely so obvious. The answers hide in innocent-looking columns, and only asking when each value arrives reveals them.

Worked example

the model that was too good

Suppose a lender builds a default model with 99 percent accuracy and prepares to celebrate.

A reviewer notices the top feature is days since last payment, which is calculated at the time the loan is closed as defaulted. The feature knows the outcome. The numbers are illustrative.

Removing it, accuracy falls to 78 percent, a realistic and still useful level.

The lesson: the first score was a warning, not a reward, and the check cost an afternoon.

The team also reruns the evaluation with a time-based split, training on earlier months and testing on later ones. The score settles near the honest level, and everyone agrees it is the number to plan with.

Illustrative case study

A fictional hospital builds a model to predict readmission and gets stunning accuracy. In live use it barely beats chance.

The cause: one input counted follow-up appointments booked, which are only booked for patients the staff already expect to return.

The invented lesson: ask when every input becomes known. A model cannot use tomorrow's information today.

After the review, the hospital adds a feature register: each input lists its source system and the time it becomes available. New models cannot enter testing until every row is signed off by someone who knows the workflow.

Common misconceptions and mistakes

Leakage is a data-quality problem. It is a design problem: the data is accurate but arrives too late to use.
A high test score means a good model. A high score is exactly what leakage produces.
Only careless teams leak. Experienced teams leak too, which is why timing checks are routine.

Business use

Treat any too-good result as suspicious until each input's timing has been checked against live use.

Insist on time-based splits for forecasting and a written list of when each feature becomes available.

Make timing review a standing step in every modeling project: a one-page table of each input, when it is known and where it comes from.

Check your understanding

Why is a surprisingly high test score a warning sign?

Answer: Leaked information makes both training and test results look excellent, and the gap appears only when the model faces real data without the leak.

Related terms

Overfitting
Feature
Validation
Time series
train-test split
target

Learn next

AI for Business Owners helps you question model results that look too good to be true: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S24] scikit-learn documentation, Glossary of Common Terms and API Elements; [S27] Hopsworks, MLOps Dictionary, Data Leakage. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Data Lakehouse Next: Data lifecycle