Can extraction occur through a black-box API?

Research demonstrates that it can for various models and access settings, though feasibility varies.

Does an extracted model contain the original training dataset?

Not necessarily. Functional imitation and training-data reconstruction are separate outcomes.

Is a model with 100% agreement on a test set an exact copy?

No. Finite evaluation establishes agreement only on the tested inputs and does not reveal identical parameters or behavior elsewhere.

AI Glossary / Statistics and probability / Intermediate

Model extraction

Definition

Model extraction is the recovery or approximation of a machine-learning model's functionality, structure or parameters through access to the model or its exposed behavior. In the security setting, an adversary may use a prediction interface to build a substitute that imitates a protected target. The result can be a functional approximation rather than an exact copy of the original weights. Authorized extraction-style evaluations can also help owners assess exposure. The term is distinct from reconstructing training records, inferring training-set membership or simply exporting a model one is entitled to access.

What it means

A prediction service necessarily reveals something about the function it computes. Repeated observations of inputs and outputs can provide information useful for learning a substitute. How much can be recovered depends on the target's complexity, accessible output detail, query budget, input coverage and prior knowledge. Confidence scores may expose more information than labels alone, but removing scores does not guarantee that imitation becomes impossible. The relevant risk is often economic or functional: a substitute may reproduce useful behavior without revealing the original implementation.

How it works

In an authorized assessment, define the target, permitted interface, query limits and recovery objective. Collect allowed input-output observations and use them to evaluate whether a substitute can approximate the target. Measure agreement on separate, representative inputs and distinguish that fidelity from accuracy against independent ground truth. Record the access assumptions, resources and domain coverage. Assess protective measures such as access controls, monitoring and output minimization in light of their effect on legitimate users. No single control should be described as a universal guarantee against all extraction strategies.

Three practical examples

Security evaluation: an organization tests how closely its own prediction service can be imitated under a defined access budget.
Service design: compare the utility and exposure implications of returning labels, scores or richer outputs.
Incident assessment: investigate suspicious query patterns and estimate whether an unauthorized substitute could reproduce valuable functionality. Similar outputs alone do not conclusively establish copying, and evidence must be interpreted carefully.

Think of it this way

Think of learning to imitate a skilled operator by observing many examples of what they do. The imitator may reproduce visible decisions without knowing the operator's exact internal process. A close match on familiar situations does not prove identical behavior everywhere, and copying a mistake still counts as imitation even though it does not improve correctness.

Worked example

In an illustrative authorized evaluation, compare a target classifier and a substitute on 100 independently labeled binary cases. They are both correct on 86 cases and both wrong on 8. The target alone is correct on 4, while the substitute alone is correct on 2. Because a binary classification has only one wrong label, the two systems agree on the 86 jointly correct plus 8 jointly wrong cases: fidelity is 94%. Target accuracy is (86+4)/100 = 90%, while substitute accuracy is (86+2)/100 = 88%. Thus 94% agreement does not mean 94% accuracy, exact parameter recovery or universal equivalence. The example also shows that a substitute can occasionally correct a target error while still being less accurate overall. These invented counts illustrate evaluation distinctions and do not describe an extraction attack against a real service.

Illustrative case study

In a fictional case, Alder Vision owns an image-classification API and commissions a controlled assessment. The team measures substitute agreement under an approved query budget and finds that ordinary cases are easier to imitate than rare classes. It reviews output detail, access policies and monitoring while checking the impact of proposed changes on legitimate customers. The resulting report describes the tested access conditions and coverage instead of claiming that one fidelity score measures every possible extraction risk.

Common misconceptions and mistakes

Extraction does not necessarily recover the original weights, architecture or training data. High agreement on one dataset does not establish equivalence across the full input space. Accuracy and fidelity are different metrics, and a substitute may reproduce the target's errors. Removing confidence scores or imposing a simple rate limit is not an absolute defense. Membership inference and model inversion concern different information goals, although an extracted substitute can sometimes support other analyses. Legitimate repeated usage should not automatically be classified as an attack solely because its query volume is high.

Business use

Businesses should identify what they are trying to protect: proprietary functionality, parameter confidentiality, training-data privacy or service economics. These objectives require different evidence and controls. Assess exposure using authorized tests with realistic access assumptions and documented evaluation data. Combine technical controls with appropriate service governance, and monitor for patterns that warrant investigation without overstating what those patterns prove. Reassess when interfaces, output detail or model families change. Security claims should describe the tested threat model and residual limitations.

Check your understanding

Why is fidelity 94% while substitute accuracy is 88% in the example?

Answer: Fidelity counts agreement with the target, including eight shared mistakes. Accuracy counts agreement with ground truth, including two cases where only the substitute is correct. The reference used for comparison determines what each metric means.

Related terms

Model inversion
Adversarial machine learning
Model stealing
substitute model
prediction API
fidelity
black-box access
membership inference
output minimization

Learn next

Study threat-model specification and the difference between functional and parameter recovery. Then examine authorized evaluation design, query-budget reporting and the utility tradeoffs of protective controls, keeping training-data privacy assessment separate.

Data privacy

Sources

Tramer and colleagues, Stealing Machine Learning Models via Prediction APIs, USENIX Security 2016: https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer . NIST AI 100-2e2025, adversarial-ML taxonomy: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf . The binary evaluation counts and fictional assessment case above are original.

Previous: Model editing Next: Model-free (reinforcement learning)