Does an attack always modify data?

No. Some attacks seek to infer or collect information without changing the target system.

Is adversarial testing itself malicious?

Authorized testing can simulate attack techniques for defensive evaluation; intent, permission and scope distinguish it from hostile activity.

Can an attack use normal-looking inputs?

Yes. A harmful objective may be pursued through interfaces and content types the system normally accepts.

AI Glossary / Governance, ethics and safety / Beginner

Attack

Definition

An attack is a deliberate attempt to compromise a system, its information or its operation. In AI, an attack may target the surrounding software and access controls, the model's training data, its inputs or outputs, or the confidentiality of information associated with it.

What it means

An AI system can fail accidentally, but an attack involves intentional adverse action. A blurred photograph that causes a mistake is not automatically an attack. A deliberately manipulated input intended to cause an unauthorized outcome may be one. Evidence of intent and mechanism matters when investigating an event.

An attack need not succeed to be an attack attempt. It also need not crash the service. An attacker may want a quiet change in one prediction, access to confidential data or an unauthorized action performed through an otherwise functioning assistant.

The useful business question is therefore broader than “Can someone break the model?” It is “What could an adversary try to make this complete system reveal, change or do?” Models operate inside workflows with identities, databases, tools and people, all of which affect the possible consequences.

How it works

Begin by identifying the asset and the property at risk: confidentiality, integrity or availability are common categories. Then specify the adversary's objective, knowledge and access. Someone who can only submit ordinary queries has different capabilities from someone who can alter a training dataset or replace a model file.

Training-time poisoning attempts to influence learning through manipulated data or model components. Inference-time evasion manipulates inputs to induce unwanted predictions. Privacy attacks seek sensitive information, while attacks on generative systems may try to redirect behavior through malicious instructions or compromised context.

These categories can overlap with ordinary cybersecurity failures. A leaked service credential can expose an AI application's database without exploiting its neural network at all. Conversely, a malicious document may influence an assistant through a content-processing path that looks legitimate at the network level.

Defensive work connects each plausible attack to a control and a way to observe outcomes. Authorized testing should use a defined scope, suitable test data and clear success criteria, so an observed failure leads to a specific repair rather than a vague claim that the system is insecure.

Three practical examples

An attacker attempts to alter labeled training records so a classifier learns a misleading pattern.
A malicious document tries to make a retrieval-based assistant treat document text as instructions for using its tools.
An adversary submits requests intended to overwhelm an AI service and prevent ordinary users from completing their work.

Think of it this way

Think of a workplace with doors, records, staff procedures and a reception desk. An intruder might force a door, forge a document or persuade someone to misuse an authorized process. Protecting only the front door misses other routes.

The analogy shows why system boundaries matter. AI attacks can exploit statistical behavior and information flows that have no exact physical counterpart, so familiar security practices need task-specific analysis.

Worked example

Separating attack success from model quality

Suppose an invented authorized evaluation contains 40 attempts to make a document assistant perform an unauthorized export. Five produce an unsafe suggested action, but a separate permission control blocks all five before export. No data leaves the test environment.

The unsafe-suggestion rate is 5 / 40 = 12.5% for this test set. The completed-export rate is 0 / 40 = 0%. Both numbers are relevant, but they measure different stages. Reporting only zero completed exports would hide the model-level weakness; reporting five successful data leaks would overstate the observed outcome.

The test does not establish the real-world frequency of attack success. Its value is that it identifies a failure path, demonstrates the tested control's effect and provides cases for evaluating a repair.

Illustrative case study

A fictional company, Alder Purchasing, introduces an assistant that reads supplier documents and prepares order drafts. During an authorized review, a document contains text asking the assistant to bypass the approval step.

The assistant proposes an action outside the intended workflow, but the purchasing service rejects it because approval is tied to the specific order version. The team records both the attempted instruction redirection and the enforced control, then improves how external text is handled.

The case illustrates layered defense. A model's willingness to propose something and a system's authority to execute it are different. Both need evaluation, and neither result should be misreported.

Common misconceptions and mistakes

Every incorrect answer proves an attack occurred. Accidental errors and distribution changes can also cause failures.
An attack matters only if the service goes offline. Confidentiality and integrity can be compromised while it remains available.
Protecting the model file protects the whole application. Data, credentials, tools and workflow permissions remain relevant.
Passing a fixed attack suite proves permanent security. Tests cover particular versions, capabilities and conditions.
A blocked attempt means there is nothing to learn. It can reveal weaknesses in earlier layers or changes in adversary behavior.

Business use

For business teams, an attack description should name the asset, attempted outcome, entry point and observed effect. That structure supports prioritization: an unauthorized draft suggestion, a blocked action and an actual data disclosure require different responses.

Connect AI security testing to ordinary system security and incident handling. Maintain the ability to limit actions, investigate relevant events and recover affected components. The practical aim is to reduce the chance and consequences of plausible attacks, with evidence about which controls work under which conditions.

Check your understanding

A model produces an incorrect answer after an ordinary user supplies an unclear question. Is “attack” the best diagnosis from that fact alone?

Answer: No. The error needs investigation. Intentional manipulation, accidental ambiguity and model limitations are different explanations requiring different evidence.

Related terms

Adversarial machine learning
Data poisoning
Prompt injection
Integrity
Availability
Threat
vulnerability
evasion
confidentiality
threat model

Learn next

Choose one AI workflow and identify what an attacker might want to reveal, change or disrupt. List the access each scenario requires and the system component that can prevent or detect it. Read threat modeling next to turn those observations into a structured assessment.

Sources

Sources: NIST CSRC glossary, attack: https://csrc.nist.gov/glossary/term/attack NIST AI 100-2e2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf Aalto Dictionary of Machine Learning, attack: https://aaltodictionaryofml.github.io/ADictML_Main.pdf

Previous: Attachment ambiguity Next: Attention