Are adversarial examples limited to neural networks?

No; the broader problem concerns machine-learning systems.

Does high confidence make an answer trustworthy?

A model can be confidently wrong on a crafted input.

Does adversarial training solve every case?

No. Evidence must be tied to the tested conditions and any claimed guarantee's assumptions.

AI Glossary / Governance, ethics and safety / Advanced

Adversarial example

Definition

An adversarial example is an input deliberately modified to cause a machine-learning model to produce an incorrect prediction or other unintended behavior. In the deployment-time setting described by NIST, the altered sample is presented to an existing model. The modification is chosen for its effect on that model, rather than being an ordinary accidental variation in the data.

What it means

A model can be highly accurate on typical inputs yet fail on inputs selected specifically to exploit its weaknesses. Some image examples involve changes that are difficult for a person to notice. However, invisibility is not a universal requirement: meaningful constraints depend on the application and what an adversary can actually change.
The correct interpretation also depends on whether the underlying task answer remains the same. If an edited picture genuinely becomes a different object, a changed prediction may be appropriate. An adversarial evaluation should state the permitted changes and justify why the expected answer is preserved.

How it works

The model has learned a relationship between inputs and outputs. A crafted modification can move an input across a decision boundary while leaving task-relevant meaning intact. A targeted test seeks a particular incorrect answer; an untargeted test seeks any incorrect answer.
Knowledge and access vary. A white-box evaluator can inspect model details, while a black-box evaluator has more restricted information, such as query results. The original fast-gradient research illustrates how input gradients can identify influential directions. In this deployment-time example, the model's trained parameters remain fixed while the input changes.
The result must be assessed under an explicit threat model: what is changeable, how much alteration is allowed, what access exists, and what outcome counts as success. A failure under one set of assumptions does not establish failure under every other set.

Three practical examples

An authorized image-classification test alters a product photograph within a stated distortion limit while keeping the product identity unchanged. The team records whether the model's answer changes and whether the modified image still satisfies the test's rules.
A document-reading team evaluates deliberately altered scans that remain legible to reviewers. It separates ordinary scanning errors from modifications selected to expose model weaknesses.
A speech-interface team tests modified recordings whose intended words remain understandable. It assesses both transcription errors and whether the downstream application takes an inappropriate action; these are related but distinct outcomes.

Think of it this way

Think of a student who recognizes familiar practice questions but can be misled by carefully chosen wording that leaves the actual question unchanged. Randomly smudging a page is a different test from selecting a change because it triggers a particular misunderstanding. Both may matter, but they measure different weaknesses.

Worked example

Use a toy classifier with two numeric inputs and score s = x1 + x2 − 1. It predicts class A when s is at least zero and class B otherwise. Suppose the correct class for a particular object remains A throughout the small measurement changes allowed in this teaching example.
For input (0.52, 0.52), the score is 0.04, so the prediction is correctly A. A deliberately selected modified input is (0.49, 0.49). Its score is −0.02, so the prediction becomes B even though the stipulated underlying class is unchanged.
Each coordinate changes by −0.03. The largest absolute coordinate change is 0.03; the Euclidean change is the square root of (0.03 squared + 0.03 squared), approximately 0.0424. A test with a maximum-coordinate allowance of 0.03 accepts this modification, including its boundary value.
The same change would not flip input (0.80, 0.80): its new score would remain 0.54. Vulnerability depends on the input as well as the model and constraint. This arithmetic shows a boundary crossing; it does not prove that a real image alteration of that size is imperceptible or preserves its true label.

Illustrative case study

Fictional Cedar Catalog uses image classification to suggest product categories. An internal evaluation begins with 200 images the model classifies correctly. Under one documented modification procedure, 30 valid altered images receive incorrect predictions. The observed success rate is therefore 30 / 200, or 15%, for this selected set and procedure.
The team reports the original sample selection, validity checks and modification limit alongside that number. It does not describe 15% as the error rate for all catalog traffic. It also keeps ordinary unmodified accuracy separate.
After a model update, it repeats the evaluation and introduces independent tests rather than assuming success on familiar examples proves broad robustness. It checks whether suspicious results reach customers or are caught by the catalog review workflow. The case illustrates how to connect a technical failure to the actual product consequence without overstating what the experiment establishes.

Common misconceptions and mistakes

Calling every wrong answer adversarial: ordinary mistakes need not involve deliberate manipulation.
Requiring every change to be invisible: constraints can involve meaning, functionality or physical feasibility instead.
Confusing deployment-time evasion with training-data poisoning: they intervene at different stages.
Treating a failed attack attempt as proof of robustness: the procedure may simply have been ineffective.
Reporting success rates without their denominator or validity rules: the resulting number is difficult to interpret.

Business use

Use adversarial examples to investigate specific failure pathways in an authorized evaluation. Define the input route, permitted manipulations and consequence of a wrong output. Compare ordinary performance with performance under the stated tests. Consider the surrounding workflow, including review and fallback behavior, because a model error and a harmful business outcome are not automatically identical.

Check your understanding

Why is the toy modified input adversarial under the stated assumptions?

Answer: It changes the prediction while the correct class remains A.

What is its largest coordinate change?

Answer: 0.03.

Why does Cedar's 15% need qualification?

Answer: It applies to initially correct samples under one specific procedure.

Related terms

Learn next

For a hypothetical classifier, write a clear expected answer and allowed-change rule before constructing a test. Record invalid modifications separately from valid examples that do or do not change predictions.

Sources

NIST AI 100-2e2025, adversarial-example terminology and deployment-time attack taxonomy. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf Goodfellow, Shlens and Szegedy, Explaining and Harnessing Adversarial Examples. https://arxiv.org/abs/1412.6572 TensorFlow, Adversarial example using FGSM, input-gradient illustration with fixed model parameters. https://www.tensorflow.org/tutorials/generative/adversarial_fgsm The two-input classifier and Cedar Catalog evaluation are original teaching illustrations.

Previous: Adversarial Domain Adaptation Next: Adversarial inverse reinforcement learning