AI Glossary / AI and machine learning / Intermediate
Label noise
Definition
Label noise is error or unreliability in the target labels used to train or evaluate a supervised learning system. It occurs when a recorded label does not accurately represent the intended target for an example.
What it means
A model learns from examples paired with answers. If some answers are wrong, inconsistent or based on a different definition, the learning signal becomes unreliable. An image may be assigned the wrong category, a support ticket may use an outdated routing label or a measurement may be matched to the wrong record.
Not all disagreement is a simple mistake. A genuinely ambiguous example may admit several defensible labels, or annotators may follow different interpretations of the task. Treating every disagreement as noise can erase important uncertainty. The intended target and annotation policy should therefore be clear before labels are corrected.
Label noise differs from noisy input features. A blurred image is an input-quality issue; a clear image marked with the wrong class is a label issue. Both can occur together, and each can affect training and evaluation differently.
How it works
Researchers describe several noise patterns. Symmetric noise changes labels in a relatively uniform way, while class-dependent noise affects some label transitions more than others. Instance-dependent noise is related to the particular example, such as difficult cases being more likely to be mislabeled.
Real workflows can also create systematic errors through weak proxies, outdated rules or inconsistent annotator instructions. A model may memorize some incorrect labels or learn a biased boundary. The effect depends on the method, noise structure and amount of useful information remaining.
Responses include improving annotation guidance, auditing suspicious examples, collecting additional judgments and using methods designed for specified noise assumptions. Automatic error detection should produce candidates for review rather than treating model disagreement as proof that a human label is wrong. A clean evaluation set is especially valuable because noisy test labels can distort conclusions about model quality.
Three practical examples
1. A ticket-routing dataset contains labels from before and after a department reorganization. The team reconciles the label definitions before training.
2. An image-classification project asks a second reviewer to inspect examples with inconsistent annotations, preserving genuine ambiguity where appropriate.
3. A researcher tests a noise-robust loss under a defined label-flipping model and checks whether its assumptions resemble the actual data process.
Think of it this way
Think of learning from an answer key containing mistakes. A diligent learner may absorb the mistakes as well as the correct answers. The limit is that machine-learning systems react differently depending on their design, and some disputed answers reflect an unclear question rather than a defective key.
Worked example
Suppose a fictional audit reviews 200 randomly sampled training labels and identifies 18 clear errors under an agreed policy. The observed error fraction is 18 / 200 = 9% in that audit sample.
Of 40 examples from class A, 10 are wrong, giving 25%. Among the remaining 160 examples, 8 are wrong, giving 5%. The overall 9% hides a much higher problem rate in one class.
The audit does not prove the exact error rate for the entire dataset, because it is a sample. It does indicate that targeted investigation of class A may be more useful than assuming every class has the same noise pattern.
Illustrative case study
A fictional delivery-support team trains a classifier using historical ticket categories. The model repeatedly disagrees with labels for one issue type. Review shows that staff had used a temporary category during a software outage.
The team documents that period, corrects labels where the intended target is clear and separates cases that remain ambiguous. It then rebuilds the evaluation set using the current category policy.
The case illustrates that label noise can come from the organization's process rather than careless annotators. Understanding how labels were produced can lead to a more reliable correction than simply replacing labels with model predictions.
Common misconceptions and mistakes
- Every model disagreement identifies a bad label. The model may be wrong or the example may be ambiguous.
- A single noise percentage describes the whole problem. Noise can vary by class, time and example difficulty.
- Robust training removes the need for good labels. Methods have assumptions and cannot recover missing truth in every situation.
Business use
Label quality affects the value of supervised AI projects and the credibility of their reported performance. Invest in a clear target definition, representative audits and a process for resolving disagreements.
Track label provenance and policy versions, particularly when operations change. Separate training-label improvement from test-set maintenance to avoid tuning the evaluation to the model. Measure whether corrections improve performance on independently reviewed examples. The objective is a trustworthy learning and evaluation signal, not forced agreement between the model and every annotator.
People also ask
Can label noise affect test data?
Yes. Incorrect evaluation labels can make a good prediction appear wrong or a bad one appear correct.
Is annotator disagreement always noise?
No. It can reflect ambiguity or a task that needs a clearer definition.
Can a model help find likely errors?
Yes, but flagged cases need appropriate review and should not be automatically relabeled solely to match the model.
Check your understanding
Why is it risky to correct every training label that disagrees with the current model?
Answer: The model can be mistaken. Automatic replacement may reinforce its existing errors and remove legitimate difficult examples.
Related terms
- Supervised learning
- Data Quality
- Weak supervision
- annotation
- class-conditional noise
- robust learning
- evaluation dataset
Learn next
Read annotation guidelines and weak supervision. Audit a small random sample of labels, then distinguish clear errors from ambiguity and policy differences before choosing a correction strategy.
More in AI and machine learning
- L1-based feature selection - L1-based feature selection uses a model penalized by the sum of absolute coefficient values to identify a smaller set of input features.
- Label - A label is the known answer attached to an example: the category, number or outcome the model is supposed to learn to predict, such as spam, churned or 42000.
- Label leakage - Label leakage is the accidental inclusion, in a model's inputs, of information that is derived from or equivalent to the answer, so the model learns to read...
- Label Propagation - Label Propagation is a semi-supervised method that spreads labels from the few cases that have them through a similarity network, so unlabeled cases inherit...
- Labeled example - A labeled example is a single training case with its known answer attached: the inputs plus the label, the atomic unit of supervised learning.
- Lagged features - Lagged features are inputs built from the past values of a series: yesterday's sales, last week's balance, the same customer a month ago, giving a model...
Sources
Natarajan, Dhillon, Ravikumar and Tewari, Learning with Noisy Labels (2013): https://papers.nips.cc/paper_files/paper/2013/hash/3871bd64012152bfb53fdf04b401193f-Abstract.html. Author-hosted paper: https://dept.stat.lsa.umich.edu/~tewaria/research/natarajan13learning.pdf. The audit figures and support-team case are fictional.
Previous: Label leakage Next: Label Propagation