Can poisoning happen without changing labels?

Yes. An attacker may manipulate the examples themselves or their inclusion while labels appear legitimate.

Does it always affect every prediction?

No. Some attacks aim at a small subset of inputs or a particular behavior.

Is prompt injection the same thing?

No. Prompt injection concerns untrusted instructions influencing a model interaction. Poisoning concerns manipulated data or learning inputs, although a system can face related or combined mechanisms.

AI Glossary / Governance, ethics and safety / Intermediate

Data poisoning

Definition

Data poisoning is deliberate interference with data used by an AI system in order to cause unwanted behavior. In its classic machine-learning sense, an attacker inserts, alters, or otherwise manipulates training data so that the learned model becomes less reliable or behaves incorrectly on selected inputs. The defining feature is adversarial manipulation, rather than an ordinary accidental data-quality problem.

What it means

Models learn from the examples they receive. If an adversary can influence those examples, they may influence what the model learns without directly editing its final parameters. The intended effect can be broad degradation, a particular wrong prediction, or a hidden behavior activated under certain conditions. Not every mislabeled example is an attack, and evidence of poor performance alone does not establish poisoning. The term is also used in discussions of manipulated retrieval collections, where the affected information is supplied at runtime rather than learned into model weights. Those mechanisms should be distinguished when investigating an incident.

How it works

A training-data attack depends on a path through which untrusted or compromised material enters collection, labeling, preparation, or model updating. The material may look ordinary enough to pass superficial checks. Its eventual effect depends on the learning algorithm, data distribution, weighting, and attacker's influence. A backdoor is one possible outcome, but poisoning can also target general accuracy or a narrow class of decisions. Defensive analysis begins with a threat model: who can modify what, at which stage, and with which objective? This is different from an evasion attack that modifies an input presented to an already trained model without changing the training process.

Three practical examples

A classifier trained from externally contributed labels needs controls against deliberate label manipulation as well as accidental annotation mistakes.
A company that periodically retrains on user feedback should distinguish trustworthy reviewed feedback from submissions designed to distort the update.
A document-answering service must protect its reference collection from malicious changes. If altered retrieved content affects answers without retraining, the investigation should describe that runtime data-integrity mechanism rather than assuming model weights were poisoned.

Think of it this way

Think of someone secretly altering practice exercises before a trainee studies them. The trainee may later apply a learned pattern that was deliberately taught incorrectly. The analogy is limited because machine-learning effects can be subtle, distributed across many examples, and difficult to trace to one item. It also does not mean that every unusual training example is malicious; legitimate rare cases must not be discarded simply because they are unfamiliar.

Worked example

A team evaluates two model versions on 1,000 ordinary test cases and a separate set of fifty cases representing a suspected targeted behavior. The earlier version answers 940 ordinary cases and forty-five targeted cases correctly: 94% and 90%. After an unreviewed data update, the later version answers 938 ordinary cases and only fifteen targeted cases correctly: 93.8% and 30%. The broad test changes by just 0.2 percentage points, while the targeted slice falls by sixty percentage points. This pattern warrants investigation, but does not prove an attack. The team should compare data versions, labels, preprocessing, and other changes under controlled conditions. The example demonstrates why a general average may miss a localized integrity problem; it does not describe how to construct poisoned data.

Illustrative case study

Fictional case: Rowan Routing retrains a message classifier from accumulated feedback. Reviewers notice an unusual concentration of label changes from one ingestion source. They quarantine the affected update, preserve the original records and processing logs, and compare a clean retraining run with the suspect version. The investigation considers both a broken labeling integration and deliberate interference. The case shows a measured response: protect the pipeline and establish evidence before assigning cause. It does not imply that a suspicious pattern alone identifies an attacker.

Common misconceptions and mistakes

Data poisoning is not synonymous with hallucination, ordinary bias, or every incorrect label. A model can fail without adversarial influence. Conversely, strong average accuracy does not rule out a targeted backdoor or narrow failure. Simple outlier removal is not a complete defense, because malicious examples may resemble legitimate data and unusual examples may be important. Another misconception is that securing the final model file secures the whole process. Collection sources, label stores, data transformations, and update approvals can all affect what the model learns.

Business use

Protect training and reference data as part of the AI supply chain. Record provenance and version history, restrict modification rights, review meaningful changes, and keep reproducible data snapshots. Use independent evaluation sets and targeted checks derived from the system's threat model. Where practical, stage updates before broad release and retain a tested rollback path. During an investigation, preserve evidence and separate observations from conclusions about intent. Defensive controls should be proportionate to the actual access paths and consequences; no single filter should be advertised as eliminating every poisoning risk.

Check your understanding

A model's overall test accuracy is almost unchanged after a data update, but performance on one important slice collapses. Can poisoning be ruled out?

Answer: No. Investigate the update and the targeted behavior. The pattern is evidence of a problem, not by itself proof of malicious intent.

Related terms

Adversarial machine learning
Backdoor
Model poisoning
Prompt injection
training-data integrity
evasion attack
data provenance
supply-chain security

Learn next

Map every route by which data or labels enter a model update. Identify the owner, modification permissions, version record, and review step for each route. Then define a small set of integrity tests that would reveal localized failures missed by the general evaluation average.

Sources

NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, 2025 edition: https://csrc.nist.gov/pubs/ai/100/2/e2025/final NIST report PDF: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf

Previous: Data point Next: Data preservation