Can leakage occur without a malicious attack?

Yes. A normal release can disclose information even if nobody has yet attempted to exploit it.

Is mutual information the same as an attack's accuracy?

No. They quantify different properties, although both can inform an assessment under suitable assumptions.

Does encryption remove output leakage?

It protects information from unauthorized reading in specified settings; an authorized recipient may still infer private facts from the decrypted output.

AI Glossary / Governance, ethics and safety / Intermediate

Privacy leakage

Definition

Privacy leakage occurs when information released by a system reveals something about protected data beyond the intended disclosure. In machine learning, outputs, representations or model artifacts can carry information about individuals or records used by the system.

What it means

Leakage need not look like a copied name or a complete document. A response may make a sensitive attribute easier to infer, reveal whether someone participated in a dataset or expose a pattern that becomes identifying when combined with other information.

The amount and meaning of leakage depend on the protected information and the recipient's prior knowledge. A fact that is already known to one recipient may be new to another. The same release can therefore create different practical risks in different contexts.

Privacy leakage is distinct from data leakage in model evaluation. The latter commonly means information from a test set or future outcome improperly entering model development. That can invalidate performance estimates without necessarily exposing personal information. The two problems can coexist, but they require different analyses.

How it works

Define the protected variable or record and the information available before and after release. Then choose a measure suited to the question. An empirical assessment may compare an attacker's inference performance with a baseline. An information-theoretic model may use mutual information to quantify statistical dependence between a release and a private variable.

Formal privacy frameworks answer different questions. Differential privacy constrains output distributions across neighboring datasets under a specified privacy unit and parameters. Its bound is not simply an observed attack-success rate or the percentage of records exposed. These measures should not be treated as interchangeable numbers.

Assess the release as part of a sequence. Several outputs may reveal more together than each appears to reveal alone. Include auxiliary information, repeated queries and downstream copies in the threat model. A low estimate or unsuccessful attack test is bounded by its methods and assumptions. The practical harm also depends on how the inferred information may be used, not only on a numerical leakage score.

Three practical examples

A model's detailed outputs make it easier to infer whether a candidate record was included in its training data.
A supposedly harmless profile representation retains information about a sensitive attribute that was removed from the original feature list.
Two aggregate reports, considered together, reveal a fact about a very small group that neither report was intended to disclose individually.

Think of it this way

Think of assembling a picture from several partially transparent sheets. Each sheet may show little on its own, but their combination can make an outline clear.

The analogy does not imply that all information combines in a simple additive way. It illustrates why privacy analysis should consider what recipients can learn from multiple releases and background knowledge together.

Worked example

A release that changes what can be inferred

Consider an invented population in which a private binary attribute is equally likely to be 0 or 1. Without other information, a recipient's best fixed guess is correct 50% of the time.

Suppose a released signal equals the private bit with probability 0.90 and flips it with probability 0.10, independently under this simplified model. Guessing the released value is then correct 90% of the time. The release has materially improved inference even though it is sometimes wrong.

In information terms, the private bit has one bit of uncertainty initially. The conditional uncertainty after observing the signal is the binary entropy of 0.10, approximately 0.469 bits, leaving about 0.531 bits of mutual information. This illustrative calculation depends on the stated model; it is not a differential-privacy guarantee or a universal measure of harm.

Illustrative case study

Fictional case: Birch Profiles removes a sensitive field before sharing an encoded customer representation with a research partner. The team initially assumes the removal resolves the disclosure concern.

An authorized assessment finds that other features still allow the specified attribute to be predicted better than an appropriate baseline. The team reviews whether the representation needs those features, what the partner can combine with it and which protection measures address the actual inference risk. It documents the limits of the test.

The scenario demonstrates residual information after a direct field is removed. It does not claim that every embedding reveals every attribute or that one classifier fully measures privacy leakage.

Common misconceptions and mistakes

Leakage requires exact reconstruction. Partial information or improved inference can also matter.
Random errors eliminate leakage. A noisy signal may still reveal substantial information.
Different privacy measures share one universal scale. Their definitions, assumptions and units differ.

Business use

Businesses should ask what a privacy assessment measures and what it leaves outside its scope. A result stated as “low leakage” is incomplete without the target information, access assumptions, method and baseline. Connect technical findings to plausible consequences for the people involved.

Control unnecessary disclosures at the source, review repeated releases and preserve appropriate access boundaries through derived artifacts. Where a formal guarantee is claimed, examine its complete specification and implementation. Where evidence is empirical, report its uncertainty and avoid turning limited test results into a claim that inference is impossible.

Check your understanding

A release guesses a private attribute correctly 90% of the time where the baseline is 50%. Does its 10% error rate establish privacy?

Answer: No. The release substantially improves inference under the stated model despite being imperfect.

Related terms

Privacy attack
Mutual information
Differential privacy
membership inference
auxiliary information
information disclosure

Learn next

Read mutual information and threat model. For one proposed release, describe what a recipient knows before seeing it, what they might learn afterwards and which measurement would address that difference.

Mutual information

Sources

The Aalto Dictionary of Machine Learning, privacy leakage: https://aaltodictionaryofml.github.io/ADictML_Main.pdf. Makhdoumi and colleagues, From the Information Bottleneck to the Privacy Funnel: https://arxiv.org/abs/1402.1774. NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.pdf. Relevant returned definitions and formulation text were reviewed. The binary-signal calculation and case are original illustrations.

Previous: Privacy funnel Next: Privacy protection