Does open source guarantee interpretability?

No; accessible computations can still be difficult to understand.

Can a black box be tested?

Yes; its behavior and performance can be evaluated.

Is a fluent rationale a verified explanation?

No; fidelity requires additional evidence.

AI Glossary / Statistics and probability / Advanced

Black box model

Definition

A black box model is a model whose internal process for producing outputs is difficult for people to understand. Users may observe inputs and outputs and measure performance without having a clear, faithful account of how the model arrives at each result. The term describes a limitation in interpretability, not a particular training algorithm or a guarantee about accuracy.

What it means

Opacity can arise from a complicated learned computation, limited access to the implementation, or both. Publishing code and parameters does not automatically make a large model understandable. Conversely, a simple undisclosed rule may be opaque to someone who can only query it.
Understanding observed behavior is useful, but it differs from establishing the mechanism behind that behavior.

How it works

A trained model maps an input representation to an output through learned parameters and computations. Analysts can evaluate it using representative examples, controlled input changes, error analysis and comparisons with simpler alternatives. When internals are available, they can also inspect intermediate computations.
A post-hoc explanation describes an already trained model. A simpler surrogate can approximate its outputs within a specified region, but agreement on tested examples does not prove equivalence everywhere. Input perturbations reveal responses to those particular changes; unrealistic combinations may say little about normal use.
Separate questions should be asked about predictive performance, explanation fidelity, stability and suitability for the intended task. Success on one dimension does not establish success on the others.

Three practical examples

Image classification: a team evaluates an accurate classifier and investigates whether changes to backgrounds alter its predictions unexpectedly.
Text routing: analysts compare a service's outputs on ordinary messages, ambiguous cases and controlled wording changes without assuming its verbal explanations reveal its internal computation.
Numeric forecasting: a team fits a simple local approximation to a complex predictor and measures how closely it agrees on separate nearby examples.

Think of it this way

Imagine a sealed instrument with controls and a display. You can learn how the display responds to many control settings, but a finite set of observations may fit several different internal designs.

Worked example

Consider an original numeric query exercise. An unknown deterministic model returns 1 for input zero, 3 for input one and 5 for input two. An analyst proposes the simple rule f(x) = 2x + 1. It matches all three observations exactly.
However, another rule, g(x) = 2x + 1 + x(x-1)(x-2), also matches them. At zero, one and two, the added product vanishes because one factor is zero. The three observations therefore cannot distinguish these two candidate mechanisms.
At input three, the first rule returns 7. The second returns 7 + 3 times 2 times 1 = 13. A new query at three can distinguish these particular candidates, but even four matching observations would not identify an unrestricted unknown function uniquely. Other rules could agree at all queried points and differ elsewhere.
Now consider a separate hypothetical model returning a score between zero and one from two binary inputs, A and B. Its stipulated outputs are score(0,0) = 0.5, score(1,0) = 0.7, score(0,1) = 0.4 and score(1,1) = 0.8.
Starting from (1,1), changing only A to zero changes the score from 0.8 to 0.4, a decrease of 0.4. Changing only B to zero changes it from 0.8 to 0.7, a decrease of 0.1. These are observed model-response differences for the specified baseline, not automatically causal effects on an external outcome.
If the analyst treats those decreases as independent and additive, they predict score(0,0) = 0.8 - 0.4 - 0.1 = 0.3. The actual stipulated score is 0.5. The interaction contrast is 0.8 - 0.4 - 0.7 + 0.5 = 0.2. The effect of changing A from zero to one is 0.2 when B is zero but 0.4 when B is one. A single context-free statement about A's contribution would miss that distinction.
Finally, imagine a separate held-out evaluation with 100 examples: 80 from category U and 20 from category V. The model is correct on 72 U examples and eight V examples. Overall accuracy is (72+8)/100 = 80 percent. Accuracy within U is 72/80 = 90 percent, while within V it is 8/20 = 40 percent.
These results show why overall accuracy, local input sensitivity and mechanism identification are different forms of evidence. The numerical exercises are original illustrations with stipulated outputs and counts. They do not show that any deployed model has these properties or that the evaluation categories are sufficient for a real application.

Illustrative case study

Fictional company Maple Routing evaluates an opaque message classifier. Its analysts assemble representative examples, record the model version and inspect errors by meaningful message categories. They also test controlled wording changes and compare a proposed explanatory summary with actual model outputs.
When the summary fails on nearby cases, the team narrows its claim rather than presenting it as the model's universal reasoning. The deployment decision considers performance, failure handling and the need for understandable explanations.

Common misconceptions and mistakes

Black box does not mean random, untestable or necessarily inaccurate. It also does not mean that accurate predictions establish a reliable explanation. A transparent model can still be wrong, and an opaque model can perform well on a particular evaluation.
Another mistake is treating local surrogate fidelity as proof of global fidelity. Feature importance depends on the method, reference distribution and question being asked. A model-generated rationale can sound coherent without being a verified description of its computation. Finally, input-output sensitivity is not by itself evidence of real-world causation.

Business use

The practical issue is whether available evidence supports the intended use. Teams can specify acceptable performance, explanation requirements and failure handling, then evaluate those separately. A useful report documents the model version, tested population, known failure patterns and the scope of any explanation. Where understanding individual decisions matters, a simpler model may merit comparison alongside more complex alternatives.

Check your understanding

Why did three queries not identify the numeric rule?

Answer: Two different functions matched all three.

Why did the additive score estimate fail?

Answer: The stipulated model contained an interaction.

Why was overall accuracy incomplete?

Answer: The two evaluation categories had very different accuracies.

Related terms

Interpretability
Explainable AI
Surrogate model
Counterfactual explanation
Feature attribution
Model evaluation
Model transparency

Learn next

Fit a simple surrogate to a known nonlinear function, then compare its agreement inside and outside the region used to fit it.

Sources

Google Machine Learning Glossary, black box model: https://developers.google.com/machine-learning/glossary#black-box-model . The query rules, score table, evaluation counts and company scenario are original illustrations.

Previous: BL (logic) Next: Black-box variational inference