Does model poisoning require access to all training data?

No. Some attacks rely on control over updates or artifacts instead.

Is a backdoor always model poisoning?

Not necessarily. Backdoor behavior can be introduced through different mechanisms, including poisoned data.

Can ordinary validation detect it?

Sometimes, but routine tests may not include the affected conditions, so threat-specific evaluation can be necessary.

AI Glossary / Governance, ethics and safety / Advanced

Model poisoning

Definition

Model poisoning is a deliberate attack that corrupts a model through control over its parameters, updates or other model-level components. Its purpose is to influence learned behavior in ways that undermine the intended system, including targeted errors or concealed backdoors.

What it means

The term is distinguished from data poisoning by the attack surface. Data poisoning manipulates training examples or related data; model poisoning acts through model control. In practice, research terminology can overlap, so an account should state what the attacker can actually modify.

A federated-learning participant, for example, may send a model update to an aggregation service. If that update is maliciously manipulated, the attack operates at the model-update level rather than requiring the participant to supply a visible centralized dataset.

Ordinary training variation and accidental implementation errors are not automatically poisoning. Evidence of unexpected behavior warrants investigation, but the word attack implies deliberate interference rather than merely poor model quality.

How it works

The attacker needs a relevant point of influence, such as a submitted update or a compromised model artifact. The manipulated component then enters a training, aggregation or deployment process that trusts it enough to affect the resulting model.

The intended effect may be broad degradation or a targeted behavior that appears only under particular conditions. Routine accuracy tests can therefore miss a problem if they do not exercise the affected cases.

Defensive work includes controlling artifact provenance and permissions, reviewing update acceptance and testing behavior under an explicit threat model. Different defenses address different assumptions; a method suitable for one aggregation process is not a universal solution. Preserve evidence that distinguishes model-level manipulation from data changes, deployment faults or ordinary distribution shift before attributing the cause.

Three practical examples

A federated-learning operator evaluates whether manipulated participant updates can distort the jointly trained model under the assumed participation rules.
A deployment team verifies that a model artifact is the approved version before loading it into a service.
A security evaluator tests whether a model behaves normally on routine inputs but fails on a separately defined set of targeted conditions.

Think of it this way

Think of a committee combining written proposals from several contributors. A contributor can influence the final document by submitting a deliberately distorted proposal, even if the committee never sees the notes used to prepare it.

The analogy illustrates influence through an update. Model aggregation is mathematical and its vulnerability depends on the protocol, so the analogy does not imply that one malicious contribution always controls the result.

Worked example

Routine accuracy can hide targeted failure

In an invented evaluation, a model correctly classifies 9,500 of 10,000 ordinary test examples, giving 95% accuracy. A separate authorized test contains 100 examples representing a suspected targeted condition, and the model misclassifies 90 of them.

The ordinary accuracy figure does not contradict the targeted failure. The two evaluations examine different input conditions. Combining them into one average would give 9,510 correct results out of 10,100, about 94.2%, which still conceals how concentrated the failures are.

These results alone do not prove poisoning. The team must investigate provenance, modifications and alternative causes. The calculation explains why a broadly acceptable average cannot by itself rule out a targeted model-level problem.

Illustrative case study

Fictional company Alder Vision accepts trained components from several project teams. One component performs well on a standard evaluation but behaves unexpectedly when a particular visual marker appears.

The security review checks the artifact's origin, approved version and integration history, while the evaluation team examines whether the marker-related behavior is reproducible. Deployment is restricted within the organization's incident process while the evidence is assessed.

The team strengthens artifact verification and its threat-model-specific tests. It avoids announcing a confirmed attack merely because an unusual pattern was found, and it avoids dismissing that pattern because the headline accuracy remained high.

Common misconceptions and mistakes

Model poisoning and data poisoning are identical. They concern different control surfaces, even though their effects can overlap.
High average accuracy rules out an attack. A targeted behavior may affect a small, important subset.
Any faulty model is poisoned. Intentional manipulation must be distinguished from mistakes and unsuitable data.
A single robust aggregation method defeats every attack. Its protection depends on assumptions and the adversary's capabilities.

Business use

Businesses should consider model poisoning when accepting external artifacts, collaborating on distributed training or allowing several parties to update a model. The key governance question is which components are trusted and why.

Maintain versioned provenance and controlled deployment paths, and align security testing with plausible control surfaces. Review third-party assurances in relation to the actual workflow rather than treating a general clean-model label as sufficient. If a problem is suspected, coordinate technical investigation and operational containment so that evidence is preserved and unsupported attribution is avoided.

Check your understanding

A model has strong ordinary-test accuracy but fails on a narrow suspicious condition. Does the ordinary score prove that model poisoning is impossible?

Answer: No. Targeted failures can coexist with strong average performance. The behavior and its cause need investigation under a clear threat model.

Related terms

Data poisoning
Federated learning
Adversarial machine learning
backdoor attack
model integrity
supply-chain security

Learn next

Read data poisoning and model integrity. Draw the path by which an external model or update reaches production. Identify each approval and verification boundary and the evidence retained about the accepted artifact.

Data poisoning

Sources

NIST AI 100-2e2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf. NIST term entry: https://csrc.nist.gov/glossary/term/model_poisoning. The numerical evaluation and fictional artifact review are original defensive illustrations, not reported attack results.

Previous: Model parameter Next: Model preference