Is a readable explanation always faithful?

No. Clarity concerns communication; faithfulness concerns whether the account accurately reflects the relevant model behavior. Both matter.

Does interpretability require a simple model?

Not always. Simple models can help, but analysis can illuminate parts of more complex systems. The scope and limitations should be explicit.

Can an explanation prove that changing a feature will improve an outcome?

Not by itself. A model relationship or hypothetical prediction change does not automatically establish a real-world causal effect.

AI Glossary / Governance, ethics and safety / Beginner

Interpretability

Definition

Interpretability is the extent to which people can understand an AI model's behavior or the basis of its outputs in terms meaningful to them. It can concern how the model generally works or why it produced a particular result. Useful interpretation should reflect the actual system, rather than merely provide a convincing story.

What it means

An accurate prediction does not necessarily tell a user how the system reached it. Interpretability helps people inspect relationships between inputs, the model's operation, and its output. This can support debugging, informed use, and scrutiny of inappropriate dependencies.

Understanding depends on the audience and the question. A developer investigating a failure may need information different from that needed by a person affected by a decision. A feature chart that is readable to a specialist may be meaningless to another user.

Interpretability and explainability are closely related, with definitions varying across sources. This entry uses interpretability broadly for human understanding of model behavior, without treating one terminology distinction as universal or implying that understanding guarantees trustworthiness.

How it works

Some models allow relatively direct inspection of their computation, such as a small rule list with clearly defined inputs. Other systems require additional analysis to describe aspects of their behavior. A local analysis addresses a particular prediction; a global analysis addresses broader patterns.

Methods might show a decision path, input contributions, response changes under selected input variations, or representative examples. Each answers a different question and has assumptions. An approximation to a complex model can be informative without perfectly describing that model everywhere.

Evaluation should check both whether people understand the information and whether it faithfully describes the relevant behavior. Users should also know the scope of the interpretation. A persuasive account of one output should not be presented as a complete explanation of the system, and a model's own fluent narrative should not automatically be accepted as a faithful record of its computation.

Three practical examples

1. A demand-planning team examines an interpretable forecast model to see how recorded seasonal indicators contribute to its predictions before investigating an unexpected estimate.

2. A document-classification team studies cases where changing a formatting feature changes the predicted category, looking for an unwanted dependence that may not transfer to new documents.

3. A service manager reviews a small decision tree's path for a routing decision, checking whether the relevant input values and thresholds were appropriate for that request.

Think of it this way

Think of a transparent recipe that shows which ingredients and steps produced a dish. It helps someone understand the process and identify where an unsuitable ingredient entered.

The analogy is limited. Many AI explanations reveal only selected relationships or approximate a complicated computation. Understanding a recipe also does not establish that the dish is nutritious or suitable for everyone. Similarly, understanding model behavior does not establish accuracy, fairness, safety, or a causal relationship in the world.

Worked example

Consider an original toy model that predicts delivery time in minutes: 10 + 2 × distance in kilometers + 5 × peak-period indicator. The indicator is 1 during the defined peak period and 0 otherwise.

For a trip of 4 kilometers during that period, the prediction is 10 + 8 + 5 = 23 minutes. The calculation shows the baseline, the distance contribution, and the peak-period contribution. For the same distance outside that period, it predicts 18 minutes.

This makes the model's arithmetic understandable. It does not prove that traveling during the peak period causes exactly five extra minutes in reality. The coefficient may reflect patterns in the training data and omitted factors.

Interpretation also depends on the input definitions: straight-line distance and road distance are not interchangeable. A clear equation with misunderstood inputs can still mislead its user about the basis of a prediction.

Illustrative case study

Fictional case: Iris Logistics uses a model to estimate preparation delays. Staff receive a chart showing the largest input contributions for each estimate. Initially, some assume that changing the largest displayed factor will reliably reduce the actual delay.

The team revises the explanation to distinguish model associations from actionable causes. It checks the chart against model behavior and asks staff to explain what conclusions they can and cannot draw from it. Operations experts separately assess which interventions are feasible.

The case is invented and claims no measured improvement. It illustrates that a visual explanation needs both technical validation and user understanding. A chart can be easy to read while inviting an inference that its method does not support.

Common misconceptions and mistakes

Interpretability is not a guarantee that a model is correct. A simple, understandable rule can consistently make inappropriate decisions. Conversely, model complexity alone does not tell us which specific behavior an analysis can illuminate.

Feature importance is not automatically causation, and a local explanation is not necessarily a global account. More detail can also reduce usefulness if it overwhelms the intended reader. The appropriate target is sufficient, faithful understanding for a defined purpose, with limitations visible alongside the explanation.

Business use

Businesses should specify what decisions an interpretation is meant to support. Debugging, challenging an outcome, checking policy alignment, and choosing an intervention require different evidence.

Test explanations with their intended users, not only the team that created them. Check whether people can identify limitations and avoid unsupported conclusions. Preserve details of the model version, input preparation, and explanation method. When the system changes, reassess whether the explanation still describes the behavior that users encounter in practice.

Check your understanding

A chart says a location feature strongly influenced a prediction. Does it establish that moving location would cause the predicted outcome to improve?

Answer: No. It describes a model relationship under the method's assumptions. Causal and practical claims about relocation require additional evidence.

Related terms

Explainable AI
Interpretable model
Transparency
Causal inference
feature importance
local explanation
global explanation
model evaluation

Learn next

Next, compare an interpretable model with a post-hoc explanation. For one prediction, state exactly what the explanation reveals, who should use it, and one conclusion that it cannot justify.

Sources

Google Machine Learning Glossary, Interpretability: https://developers.google.com/machine-learning/glossary#interpretability NISTIR 8312, Four Principles of Explainable Artificial Intelligence: https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8312.pdf Relevant definitions and explanation principles were read. Terminology varies across sources; the calculation and scenarios are original.

Previous: Interpolator Next: Interpretable model