Does label shift mean incorrect labels?

No. Label errors concern wrong annotations. Label shift concerns changed class proportions under a stability assumption.

Can accuracy change without retraining?

Yes. A different class mixture can change aggregate performance measures even if within-class behavior remains stable.

Can label shift always be detected without new labels?

No. Unlabeled estimation methods require assumptions and sufficient information from the reference model or data. They cannot establish arbitrary distribution changes automatically.

AI Glossary / Governance, ethics and safety / Intermediate

Label shift

Definition

Label shift is a change in how common the target classes are between datasets or operating settings, while the distribution of inputs within each class remains unchanged. In probability notation, the class proportions P(Y) change but P(X|Y) stays the same. It is a specific distribution-shift assumption, not simply another name for any change in data.

What it means

A classifier may encounter more examples of one category after deployment than it saw during training. Under label shift, the kinds of examples belonging to each category remain distributed as before; what changes is their mixture.

For instance, a document-routing system might receive a larger proportion of delivery questions during a particular period. Calling this label shift assumes that delivery questions and other categories retain their respective input patterns. If the wording within a category also changes substantially, the pure label-shift assumption may fail.

The term does not mean that someone renamed categories or corrected annotations. It concerns the distribution of the true target labels, not merely the frequency with which a model predicts them.

How it works

Class proportions influence the overall mixture of inputs and the interpretation of predictions. Even if a fixed classifier behaves similarly within each true class, the proportion of its alerts that are correct can change when the classes become more or less common.

When reliable recent labels exist, teams can compare class frequencies while accounting for sampling and uncertainty. Some methods estimate changed class proportions using unlabeled deployment data together with a classifier's behavior measured on labeled reference data. Such estimation depends on assumptions and mathematical conditions; it is not a general solution for arbitrary drift.

Possible responses include adjusting estimated probabilities or weighting data for the new class mixture. Before applying a correction, check whether class-conditional input patterns are plausibly stable and whether relevant classes were represented in the reference data. A correction based on the wrong shift model can make performance worse.

Three practical examples

1. A service desk receives a greater share of billing questions after a billing cycle, while the language patterns within each request category remain broadly stable in the simplified label-shift scenario.

2. A quality-control classifier encounters a different proportion of defective items after a process change, assuming the observed characteristics within defective and nondefective classes remain distributed as before.

3. A document archive imports another collection with a different mixture of document types, while each type's input characteristics are sufficiently similar for label-shift analysis to be considered.

Think of it this way

Think of a jar containing two types of sweets. The recipe for each type stays the same, but the jar's proportions change from mostly one type to an even mixture.

The analogy is limited. Real input distributions are complex, and stability within classes is an assumption requiring evidence. If a sweet's recipe changes as well as the mixture, the example no longer represents pure label shift. Similarly, a new writing style or sensor process can introduce changes beyond class proportions.

Worked example

Consider an original toy defect detector with an 80% detection rate for defective items and a 10% false-alert rate for nondefective items. Initially, 100 of 1,000 items are defective. Expected alerts are 80 true alerts plus 90 false alerts, so 80 / 170, about 47.1%, of alerts identify defects.

Later, suppose 300 of 1,000 items are defective, while the detector's within-class rates remain unchanged. Expected alerts are now 240 true alerts plus 70 false alerts. The confirmed-defect share among alerts becomes 240 / 310, about 77.4%.

The detector did not improve its within-class detection or false-alert rates in this example. The class mixture changed. These are expected counts under the stated assumptions, not a guarantee for every sample.

The example also shows why a rise in alert precision does not, by itself, prove that a model update improved discrimination.

Illustrative case study

Fictional case: Maple Support notices that a routing dashboard produces many more account-access classifications after a login-system change. The team initially considers adjusting the model for label shift.

Reviewing recent labeled requests reveals two questions: whether account-access requests are genuinely more common and whether their language has changed because users now describe new error messages. The second possibility challenges the assumption that inputs within that class are stable.

The team evaluates the changes before selecting an adaptation method. This case is invented. It illustrates why identifying a plausible increase in class frequency is only the beginning of a label-shift assessment, especially when the event causing that increase may also change the nature of examples.

Common misconceptions and mistakes

Predicted class frequencies are not the same as true class frequencies. A model can produce more positive predictions because its errors changed, because inputs changed within classes, or because the true class mixture changed.

Label shift also differs from covariate shift, which conventionally assumes changing input frequencies with a stable conditional label relationship P(Y|X). These are different simplifying assumptions. Neither should be selected solely because its correction method is convenient. Real deployments can contain several changes at once and may fit neither assumption well.

Business use

Businesses should monitor class prevalence alongside appropriate performance measures and review the events that could change their data. Keep enough trustworthy labeled evaluation data to investigate whether apparent changes reflect reality or model error.

When considering label-shift correction, document the stable-within-class assumption, estimation method, uncertainty, and validation results. Check downstream thresholds and workload effects as well as predictive measures. A larger number of genuine cases can alter staffing needs even when the classifier itself behaves exactly as before.

Check your understanding

Alert precision rises after defects become more common, while detection and false-alert rates stay fixed. Must the detector have improved?

Answer: No. A changed class mixture can explain the precision increase, as the worked example demonstrates. Model improvement requires a suitable comparison beyond that aggregate change.

Related terms

Prior probability
Concept drift
Precision
Calibration
Distribution shift
class imbalance
covariate shift
model monitoring

Learn next

Next, compare label shift with covariate shift. For a deployed classifier, identify what evidence would support stable input patterns within each class and what observation would challenge that assumption.

Sources

Lipton, Wang, and Smola, Detecting and Correcting for Label Shift with Black Box Predictors: https://proceedings.mlr.press/v80/lipton18a.html The original paper's primary abstract was read, including its shift assumption and estimation conditions. The numerical derivation and cases are original; no deployment-specific correction is claimed.

Previous: Label ranking average precision Next: Label space