Can an attack use only outputs?

Some methods can, depending on the model, output detail and background information.

Does model accuracy establish privacy?

No. Predictive performance and information leakage are different properties.

Is a privacy attack always an external intrusion?

No. An authorized recipient or participant may also have access that enables unintended inference.

AI Glossary / Governance, ethics and safety / Intermediate

Privacy attack

Definition

A privacy attack is an attempt to learn protected information by exploiting a system's data, behavior or accessible outputs. In machine learning, it can target whether a record was used in training, private attributes of individuals or information reconstructable from a trained model.

What it means

A model can reveal more than its intended prediction. Its responses, confidence values or other accessible information may provide clues about the data from which it learned. An attacker may combine those clues with information already available from elsewhere.

Different attacks ask different questions. Membership inference asks whether a particular example was in the training set. Attribute inference seeks an unknown attribute, while reconstruction aims to recover information resembling or corresponding to underlying data. Success at one task does not automatically establish success at another.

This entry concerns machine-learning privacy attacks, including the model-access sense used by the source glossary. Ordinary theft of a database is also a privacy threat, but it does not require exploiting a learned model. Defensive evaluation should keep these mechanisms distinct so that mitigations address the actual exposure.

How it works

A privacy assessment defines the attacker's objective, access and prior knowledge. Access might be limited to prediction queries or might include model parameters, updates or internal representations. These differences can substantially change what information is available and what a test result means.

The evaluator then measures whether the available information improves inference about the protected target under a stated test design. Appropriate baselines, realistic class proportions and false-positive rates matter. An attack that appears accurate on a balanced demonstration may produce many incorrect accusations when the target condition is rare.

Defenses can include limiting unnecessary outputs, controlling access, reducing risky data use and applying suitable privacy-preserving learning techniques. None should be described as universal protection without its assumptions and evidence. A failed attack test establishes only that the tested method did not succeed under those conditions; it does not prove that no attack exists. Authorized testing should avoid exposing real people's information unnecessarily.

Three practical examples

A privacy review tests whether an external prediction interface reveals evidence that a particular record was included in model training.
A research team evaluates whether shared model updates expose information about the private examples used to compute them.
A developer checks whether a generative model reproduces restricted training content under an authorized evaluation protocol.

Think of it this way

Think of someone inferring a confidential conversation from clues left in a later public summary. They may learn something without obtaining the original recording, particularly if they already know part of the context.

The analogy does not imply that every model output leaks a secret or that inference is always accurate. It illustrates why protecting the original data file is not the only privacy boundary.

Worked example

Why false positives matter

Consider an invented membership-inference evaluation with 10,000 candidate records, of which 100 were actually in training. An attack detects 70 of those members, a 70% true-positive rate. It also flags 1% of the 9,900 nonmembers, producing 99 false positives.

The attack makes 169 positive membership claims in total, but only 70 are correct. Its precision is approximately 41.4%. The apparently low 1% false-positive rate still creates more false claims than true claims because membership is rare in this scenario.

The example does not dismiss the risk: 70 correct inferences may still matter. It shows why reporting only a detection rate or balanced accuracy leaves the privacy assessment incomplete.

Illustrative case study

Fictional case: Alder Analytics plans to expose detailed confidence scores from a model trained on customer records. Before release, an authorized review compares the proposed interface with a more limited output format under a defined privacy test.

The team documents what the test can and cannot infer, including the attacker's assumed background information. It considers changes to the interface and training process, then evaluates utility and residual risks. It does not treat the absence of a successful demonstration as a certificate of privacy.

The case illustrates a defensive review process. It provides no attack instructions against a live service and makes no claim about a measured mitigation result.

Common misconceptions and mistakes

An attacker must download the training database. Some attacks exploit model access or outputs instead.
Membership inference reconstructs the complete record. Learning whether a record participated is a distinct objective.
One unsuccessful test proves protection. Other methods, access conditions or background knowledge may produce different results.

Business use

Businesses should include model-related privacy risks in their threat assessment and procurement questions. Ask which protected information is at issue, what access a potential attacker would have and how the claimed protection was evaluated. Distinguish demonstrations from guarantees and theoretical assumptions from deployment conditions.

Consider the full release surface: model weights, prediction interfaces, logs, explanations and shared updates can expose different information. Match controls to the threat and monitor changes. If testing uses sensitive records, maintain an authorized scope and appropriate handling of any information revealed during the assessment.

Check your understanding

An evaluation shows that an attacker can identify some training members but cannot reconstruct their full records. Is the privacy concern eliminated?

Answer: No. Membership itself can reveal information in context. Reconstruction is a separate capability and is not required for every privacy harm.

Related terms

Model inversion
Privacy leakage
Differential privacy
Membership inference
attribute inference
threat model

Learn next

Read membership inference and threat model. For one AI release, specify the protected information, the recipient's access and background knowledge, and the evidence needed to evaluate unintended inference.

Background knowledge

Sources

The Aalto Dictionary of Machine Learning, privacy attack and membership inference: https://aaltodictionaryofml.github.io/ADictML_Main.pdf. NIST AI 100-2e2025, Adversarial Machine Learning: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf. Relevant returned taxonomy and definition text were reviewed. The base-rate calculation and fictional review are original illustrations, not reported attack performance.

Previous: Privacy Next: Privacy-by-design