Is it only about images with tiny pixel changes?

No. It includes several attack surfaces across predictive and generative AI systems.

Is data poisoning the same as ordinary bad data?

Poisoning involves deliberate influence; accidental data errors may produce similar symptoms but have a different cause.

Can a model be robust to one attack and vulnerable to another?

Yes. Robustness claims must specify the attack capabilities and evaluation conditions.

AI Glossary / AI and machine learning / Intermediate

Adversarial machine learning

Definition

Adversarial machine learning studies deliberate attempts to manipulate machine-learning systems and methods for evaluating or reducing those risks. Attacks can target training data, model inputs, model information or the surrounding AI workflow.

What it means

Ordinary model errors can arise from noise, unfamiliar conditions or incomplete training data. Adversarial settings add an actor intentionally trying to cause an unwanted outcome.

The actor's access matters. Someone able to alter training records poses a different threat from someone who can only submit inputs to a deployed model. The desired outcome also varies, from wrong predictions to exposure of information.

Examples include evasion through crafted inputs, poisoning of learning data and attempts to extract information about a model or its data. Generative systems introduce related threats involving instructions embedded in untrusted material.

For business readers, the term is a reminder that average accuracy on ordinary examples is not a complete security assessment. A model operates within a system of data sources, interfaces and permissions that also need examination.

How it works

A defensive assessment begins with a threat model: what an adversary can observe, modify or query, and what outcome the organization needs to prevent.

The team designs authorized tests that reflect those capabilities. It measures the complete workflow, because an attack may exploit data ingestion, retrieval or tool use rather than only the model's prediction function.

Mitigations are matched to the threat. They may include protecting data provenance, restricting permissions, validating inputs and outputs, monitoring unusual activity and training or evaluating for relevant adversarial conditions.

No single test establishes universal robustness. The team documents the tested scope, remaining uncertainty and operational response. Defenses should also be checked for their effect on legitimate users and normal task performance.

Three practical examples

Content classification: evaluate whether deliberately altered inputs cause a model to miss material it is intended to identify.
Training-data management: review whether untrusted contributors could influence labels or records used in a future model update.
Retrieval assistants: test whether instructions embedded in retrieved content improperly redirect the assistant's actions or disclosures.

Think of it this way

Think of testing a lock against someone trying to defeat it, rather than only checking that it opens with the correct key. The evaluation changes because the conditions are intentionally hostile.

The analogy has a limit. AI systems have many interacting components and probabilistic behaviors. Passing one set of attacks does not prove that every future manipulation will fail, so defense requires layered controls and continued evaluation.

Worked example

Interpreting a scoped robustness test

Suppose an invented authorized evaluation contains 100 ordinary examples and 100 modified examples produced under a documented threat model. A classifier is correct on 95 ordinary examples and 70 modified examples.

The measured accuracies are 95% and 70% for those collections. The difference suggests a weakness under the tested modifications, but it does not quantify every possible attack or the likelihood of an incident in production.

If a mitigation raises modified-example accuracy to 85% while reducing ordinary accuracy to 90%, the team has a trade-off to assess. It should inspect which errors changed and whether the test collection represents the actual threat.

These figures demonstrate evaluation reporting. They are not a benchmark, an attack recipe or evidence about any deployed system.

Illustrative case study

A fictional company gives a document assistant access to internal records and an external search tool. During an authorized test, retrieved text contains instructions unrelated to the user's request. The assistant treats that text as operational guidance.

The team strengthens the separation between source content and trusted instructions, limits tool permissions and tests whether the revised workflow still completes legitimate tasks. It records failures as system-level findings rather than blaming only the language model.

The lesson is that adversarial robustness includes the surrounding workflow. Useful defenses constrain what untrusted information can cause the system to do.

Common misconceptions and mistakes

High ordinary accuracy means the model is secure. Adversarial behavior requires separate evaluation under a defined threat model.
Adversarial training solves every attack. Its value depends on the threats represented and the complete system design.
Every unusual input is malicious. Many unusual cases are legitimate; controls should distinguish evidence of manipulation from ordinary variation where possible.

Business use

Adversarial machine learning informs AI security testing, supplier review and deployment design. It helps teams ask who can influence the system and what safeguards limit the consequences.

Request a clear threat model, authorized test evidence and a response plan for detected weaknesses. Match access controls and monitoring to the model's actual role. Avoid broad claims such as attack-proof when the evidence covers only particular input types, permissions or test methods.

Check your understanding

A model passes a test where attackers can only submit inputs. Does that establish protection against someone who can modify its training data?

Answer: No. Those are different threat capabilities. Training-data manipulation needs its own analysis and controls.

Related terms

Learn next

Read threat model and prompt injection. Map the inputs, data sources and tools in one AI workflow, then identify which parties can influence each surface and what consequences are possible.

Prompt injection

Sources

NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, 2025: https://www.nist.gov/publications/adversarial-machine-learning-taxonomy-and-terminology-attacks-and-mitigations-0. Primary taxonomy record consulted. Evaluation counts and company case are original defensive teaching examples.

Previous: Adversarial loss Next: Adversarial search