Is skew the same as data drift?

No. Drift is one possible contributor; processing mismatches and feedback can also matter.

Can skew happen when the model file is identical?

Yes. Different input values or interpretations can change its predictions.

Does sharing code eliminate it?

It reduces some risks but does not ensure matching data snapshots, populations, or feature availability.

AI Glossary / MLOps and deployment / Intermediate

Training-serving skew

Definition

Training-serving skew is a difference between how a model behaves or performs in its training and evaluation setting and how the same model behaves or performs when used to serve predictions. It can arise from inconsistent data processing, changing data, or feedback from the deployed system. Diagnosing it requires separating implementation mismatches from statistical differences between datasets and time periods.

What it means

A model learns relationships within a particular representation of its inputs. If production supplies different units, missing-value rules, category mappings, or feature definitions, the model may receive values that do not mean what they meant during training.

Even with identical processing code, the population or available information can change. A historical feature might use a value that was only finalized after the prediction would have been made. A ranking system can also influence which interactions become future training examples. The term therefore has a broader meaning than a simple difference between two code paths.

How it works

Begin with a matched comparison: use the same model version and the same logical input event in both environments. Compare raw inputs, derived features, predictions, and output interpretation. This helps locate engineering discrepancies without mixing them with differences in the sampled population.

Then compare performance across training, held-out, later, and live data using a consistent metric and label definition. Ordinary generalization gaps can occur without an implementation error. A large live drop warrants investigation but does not by itself identify the cause.

Version fitted transformations with the model, preserve feature definitions, and use data as it was available at prediction time. Where appropriate, retain representative serving-time feature records for consistency checks under the application's data-handling policy. Delayed labels must be joined to the correct historical predictions before conclusions about quality are drawn.

Three practical examples

1. Scaling mismatch: a model trained on standardized measurements receives raw measurements during serving.

2. Historical information mismatch: training examples use a finalized status that production could not yet know at the moment of prediction.

3. Feedback-driven data change: a recommendation system affects which items users see, altering which outcomes are subsequently observed and used for training.

Think of it this way

Think of learning to read a map in kilometers and then receiving distances in miles without being told. The numbers still look reasonable, but their interpretation has changed. The analogy captures a representation mismatch; it does not cover every source of skew, such as a genuinely changing population or feedback caused by the system's own decisions.

Worked example

Suppose training standardizes a feature using z = (x − 50)/10, and the fitted prediction rule is y_hat = 10 + 2z. At raw input x = 70, the correct transformed value is 2, so the prediction is 14.

If serving mistakenly passes raw 70 into the slot intended for z, the model returns 10 + 2 × 70 = 150. The model coefficients have not changed. The discrepancy comes from a different input meaning.

For raw values 50, 60, and 70, the correct predictions are 10, 12, and 14. The incorrect serving path gives 110, 130, and 150. Their absolute differences are 100, 118, and 136, averaging 118. This is a prediction-consistency comparison, not a measured error against real outcome labels.

Now assume both paths correctly apply the scaler, but the training replay retrieves a feature value of 70 while the original serving-time snapshot contained 60. The resulting predictions are 14 and 12. Shared code alone cannot eliminate this discrepancy; the compared feature snapshots must represent the same information time.

Finally, if 20 of 200 paired records differ in a checked feature, the observed mismatch rate is 10%. That count helps locate an issue but does not measure its predictive impact: some differences may be harmless while a few may be consequential. These are invented, hand-checked diagnostic examples.

Illustrative case study

Fictional case: Spruce Routing sees a drop after releasing a message-classification model. Its training and serving teams initially compare aggregate accuracy on different message collections and blame model drift.

They replay a controlled sample through both paths and discover different handling of an unknown category. After aligning the mapping, they repeat the paired check. They then evaluate later labeled messages to assess any remaining distribution change. This separates a correctable implementation discrepancy from possible changes in the task. The scenario does not claim a particular recovered accuracy or prove that all deployment issues were resolved.

Common misconceptions and mistakes

Not every training-to-live performance difference is a bug. Overfitting, sampling variability, population changes, and outcome delays can contribute. Conversely, equal aggregate metrics do not prove that individual predictions agree: discrepancies can cancel in an average.

Retraining is not a universal fix. If production still uses an incompatible transformation, a new model can inherit the same failure. A shared feature store also does not automatically guarantee matching time snapshots or prevent future information from entering historical training examples.

For stochastic models or different numerical runtimes, exact output equality may be inappropriate. Define tolerances or distributional checks that match the intended behavior while still exposing material inconsistencies.

Business use

Treat skew monitoring as part of the model's operating design. Specify which features can be compared, how often representative checks run, and which discrepancies warrant investigation. Track model, transformation, and feature-definition versions together.

Separate service health from model quality. A fully available endpoint can silently serve differently encoded inputs, while a quality metric may arrive only after outcomes become known. Early input-consistency checks and later labeled evaluations provide complementary evidence.

Check your understanding

Two paths use the same scaler and model but retrieve a customer's feature from different dates. Is identical code enough for a fair consistency test?

Answer: No. The input information must also match the relevant prediction time and event.

Related terms

Learn next

build a paired feature-and-prediction comparison, then distinguish implementation consistency from evaluation on genuinely later data.

Sources

Primary sources: Google's Machine Learning Glossary: https://developers.google.com/machine-learning/glossary#training-serving-skew ; Martin Zinkevich, Rules of Machine Learning, training-serving skew and Rules 29–37: https://developers.google.com/machine-learning/guides/rules-of-ml . Relevant passages were read. The article retains the broad source meaning while separating engineering and statistical causes. Examples are original.

Previous: Training Pipeline Next: Training set