Must an explanation be written in words?

No. A diagram, rule trace, comparison, or example can explain something when it is meaningful to its audience.

Can a correct prediction have a poor explanation?

Yes. The output may be correct while the supplied reason is inaccurate or unhelpful.

Does an understandable explanation prove a system is trustworthy?

No. It contributes relevant evidence, but reliability, safety, fairness, and other concerns require their own assessment.

AI Glossary / Governance, ethics and safety / Intermediate

Explanation

Definition

An explanation is an account that helps someone understand an AI system, its output, or a relevant aspect of its behavior. It may take the form of reasons, rules, examples, feature contributions, or a description of how an outcome was produced. A useful AI explanation must be appropriate to its audience and supported by evidence; a plausible story alone is insufficient.

What it means

The question being explained determines what information matters. A developer asking why a model failed may need evidence about a data-processing error. A customer asking why a request was rejected may need the actual decision factors and a way to correct inaccurate information. A manager asking how a system generally works may need a broader description. These are different explanation tasks. Giving the same technical chart to everyone can leave important questions unanswered even if the chart is accurate. Explanations should clarify their scope and distinguish the model's contribution from subsequent human or organizational decisions.

How it works

An explanation can come directly from an understandable decision procedure or from a separate method that analyzes a more complex model. A rule trace reports which conditions were applied. A feature attribution allocates a prediction relative to a reference under particular assumptions. A counterfactual describes a changed input associated with a different output, subject to feasibility and causal limitations. The NIST principles distinguish providing an explanation, making it meaningful to its audience, ensuring that it accurately reflects the system's process, and recognizing knowledge limits. These requirements are connected but separate: clear wording cannot compensate for an inaccurate account.

Three practical examples

A scheduling assistant shows which explicit availability and capacity rules prevented a proposed appointment.
A document classifier presents the passages that an attribution method associates with a category, with a note about the method's limitations.
A service desk explains that a request was paused because a required document was missing, identifies the document, and provides a route to correct the record. The explanation should reflect the actual workflow rather than invent a more reassuring rationale.

Think of it this way

Think of an explanation as a map prepared for a particular journey. It should show the relevant route and landmarks without pretending to include every detail of the terrain. The analogy has limits: an AI explanation may be an approximation derived from observations rather than a faithful map of all internal computation. Its omissions and assumptions need attention, especially when someone may act on it.

Worked example

Consider a fictional system with an explicit rule: send a request for review if its amount exceeds 500 units or a required field is missing. A request has an amount of 420 and is missing the required field. The accurate rule-based explanation is that the missing field triggered review; the amount threshold did not. If the amount were 620 and the field were still missing, both conditions would apply. Reporting only the amount would be incomplete if the person expects lowering it to prevent review. The worked example shows why an explanation should identify relevant conditions and avoid implying sufficient recourse from a partial reason. It also separates explaining the rule from defending it: accurately stating a threshold does not demonstrate that the threshold is justified or appropriate.

Illustrative case study

Fictional case: Willow Support deploys a language model to turn decision logs into readable explanations. Reviewers find that some drafts add sympathetic but unsupported reasons. They restrict the generator to recorded factors, require it to flag missing evidence, and test whether readers understand how to challenge an error. This invented case illustrates the difference between improving presentation and establishing factual grounding. A natural-language explanation remains a claim that must be checked against the system and the decision record.

Common misconceptions and mistakes

An explanation is not automatically a justification. It can describe how a harmful or incorrect outcome arose without making that outcome acceptable. It is not necessarily a causal account: model associations and attributions alone do not establish real-world causes. Longer explanations are not always better, because unnecessary detail can conceal the information a person needs. A model's self-reported reasoning should not be assumed to faithfully reveal its internal computation. Likewise, presenting a technical score without defining its baseline or units may create an appearance of precision while failing to explain the outcome.

Business use

Design explanations around concrete user tasks. Ask whether the person can identify an incorrect input, recognize a limitation, understand an applicable rule, or seek review. Keep an auditable connection between the explanation and the relevant model, data, and decision version. Evaluate accuracy separately from readability and user satisfaction. Where an explanation is approximate, say what was approximated and what the account cannot establish. Provide layered detail when appropriate, allowing a short practical account with access to supporting evidence. Avoid promising that disclosure alone resolves fairness, privacy, or accountability concerns.

Check your understanding

A request triggered review for two independent reasons, but its explanation names only one and implies that removing it guarantees approval. What is the problem?

Answer: The account is misleading about the decision conditions and possible recourse. It should disclose the relevant remaining condition and avoid guaranteeing an outcome that does not follow.

Related terms

Learn next

Choose a hypothetical automated decision and write separate explanations for its affected person and its technical reviewer. Check each sentence against the decision record. Mark which statements describe the system, which justify policy, and which would require additional causal evidence.

Sources

NIST, Four Principles of Explainable Artificial Intelligence: https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8312.pdf NIST AI RMF, characteristics of trustworthy AI: https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/ Ribeiro, Singh, and Guestrin, original local-explanation research: https://arxiv.org/abs/1602.04938

Previous: Explainer Next: Explanation-based learning