Does global normalization guarantee correct labels?

No. It addresses this particular mechanism but does not guarantee accurate features, sufficient data or successful optimization.

Is label bias the same as exposure bias?

No. Exposure bias concerns differences between training and prediction histories, while label bias concerns the effects of local normalization.

Why is a single outgoing transition revealing?

Its conditional probability must be one, so changing its finite score alone cannot express that the observation fits badly.

AI Glossary / Probabilistic machine learning / Advanced

Label bias problem

Definition

The label bias problem is a limitation of certain locally normalized sequence models: each state's outgoing transition probabilities must sum to one, which can remove evidence about how well that state fits the observation. This can favor paths through states with restrictive or low-uncertainty transition patterns.

What it means

Sequence models assign a connected set of labels, such as a grammatical tag for each word. A locally normalized model chooses a distribution over the next label separately for each preceding state and observation context.

That normalization makes alternatives compete within a state. It does not directly let all outgoing options from a poorly supported state receive little total probability. Even if every continuation is implausible, their probabilities must collectively sum to one.

This is a technical use of bias. It is distinct from biased human annotations, demographic unfairness and class imbalance. The classic discussion concerns maximum entropy Markov models and related conditional sequence architectures.

How it works

Suppose a state gives each allowed outgoing transition a score. A local softmax exponentiates those scores and divides by their sum. Adding the same constant to every outgoing score changes none of the resulting probabilities.

Therefore, an observation-dependent penalty shared by all continuations from that state cancels during normalization. A state with only one allowed continuation assigns it probability one regardless of that transition's finite score.

A globally normalized conditional random field instead compares complete sequence scores through a shared normalizer. This allows evidence to affect the relative support for entire paths. It removes this particular local-normalization mechanism, although feature limitations, poor training and other modeling problems can still cause errors.

Three practical examples

A part-of-speech tagging study examines whether restrictive transition patterns overwhelm evidence from the current word.
A document-segmentation team compares a locally normalized sequence model with a globally normalized model on ambiguous boundaries.
A researcher builds a tiny transition graph to distinguish model-scoring limitations from search errors in a decoder.

Think of it this way

Think of two committees that must each distribute their entire budget among their own proposals. One committee may have only weak proposals, but its total spending still cannot fall below its allotted budget.

The analogy concerns within-group normalization, not human favouritism. A sequence model's probabilities are mathematical quantities, and a committee with fewer proposals is not automatically preferred in every real decision. The issue is what evidence the scoring system preserves.

Worked example

Watching a common penalty disappear

Consider an original two-step graph. The initial transition enters state A with probability 0.55 or state B with probability 0.45. A has one allowed continuation, while B has two.

For the next observation, A's only outgoing log score is −8. After local normalization its probability is exp(−8)/exp(−8) = 1. B's outgoing log scores are log 9 and 0, giving probabilities 0.9 and 0.1.

The best complete path through A has probability 0.55 times 1 = 0.55. The best through B has probability 0.45 times 0.9 = 0.405. The locally normalized model prefers A's path even though the finite penalty −8 cannot reduce its only outgoing probability.

To illustrate a different scoring system, retain initial factors 0.55 and 0.45 but combine them with unnormalized second-step weights. The complete weights become 0.55 exp(−8), 0.45 times 9 and 0.45 times 1. Globally normalizing those three weights strongly favors B's first continuation.

This is a comparison of deliberately specified toy scores, not a valid procedure for converting every trained local model into a trained global model. It isolates the information that local normalization can discard.

Illustrative case study

Fictional case: diagnosing an extraction error

A fictional team labels sections of service reports. Its sequence model often persists along a restrictive label pattern even when a later sentence strongly suggests a different interpretation.

The team checks the allowed transitions and compares complete path probabilities on a small set of reviewed examples. It separately checks annotation quality and decoder search, rather than attributing every error to label bias.

It then trains a globally normalized baseline with comparable features and evaluates complete label sequences on held-out reports. Any improvement is measured; it is not assumed from the architectural change alone.

Common misconceptions and mistakes

Label bias is not simply a preference for frequent labels. It arises from how a model normalizes and propagates scores, although label frequencies may influence learned parameters.

Using a wider beam or exact decoding does not automatically fix it. Better search can find the model's highest-scoring path more reliably, but it cannot restore information removed by the scoring rule.

Nor does every use of softmax imply the same problem. The surrounding factorization and dependencies matter. A sufficiently expressive conditional model with access to the relevant context can behave differently from a restricted local model. Avoid treating label bias as a blanket proof that all autoregressive modeling is inadequate.

Business use

This concept is useful when reviewing structured prediction systems whose errors seem tied to transition topology or locally certain states. It provides a concrete question: can evidence lower the total support for a path, or only redistribute support among its continuations?

Inspect scores, normalization and available context before choosing a remedy. Compare local and global models using matched data and suitable features, and include computational cost in the decision. Keep search errors, training-data problems and this normalization effect separate in the evaluation report so that a proposed fix addresses the observed cause.

Check your understanding

If A's only outgoing log score in the example changes from −8 to −80, what happens to its locally normalized transition probability?

Answer: It remains one in exact arithmetic. The numerator and denominator change together. A numerically stable implementation must avoid underflow obscuring that fact.

Related terms

Maximum-entropy Markov model
Conditional random field
Sequence labeling
Softmax
local normalization
global normalization
exposure bias

Learn next

Compare local and global normalizers on the same tiny graph, then separate the model's preferred path from the path found by a limited search procedure.

Sources

Primary technical exposition: Awni Hannun, The Label Bias Problem: https://awni.github.io/label-bias/

Primary teaching material: Eric Xing, Conditional Random Fields: https://www.cs.cmu.edu/~epxing/Class/10708-17/slides/lecture9-CRF.pdf

The numerical graph and service-report case are original teaching examples.

Previous: Label Next: Label leakage