AI Glossary / Causal inference and experimentation / Intermediate

Counterfactual explanation

Definition

A counterfactual explanation describes how an input could be changed so that a model would produce a different specified output. It explains a particular prediction through an alternative input scenario, often seeking small, understandable changes while respecting constraints on what can realistically be changed.

What it means

Instead of only saying which features contributed to a prediction, a counterfactual explanation asks what would need to differ for another result. For example, a scheduling model might classify a proposed plan as feasible if a start time were later and an unavailable resource were replaced.
The explanation is usually local to one input and one model version. It does not reveal every aspect of the model, and several different alternatives may change the output.
Crucially, a model-level counterfactual is not automatically a real-world causal recommendation. Changing a recorded feature may not be feasible, may affect other features, or may fail to produce the promised outcome after the system changes.

How it works

Choose the original input, target output and constraints. Search for an alternative input that reaches the target while limiting some measure of change, such as distance or number of modified features. Add requirements for plausibility, immutable attributes, permitted actions and dependencies between variables.
Validate the proposed alternative by evaluating it with the actual model. Where the explanation is intended as practical recourse, assess action costs, causal dependencies and robustness to small changes or model updates. Explain which values changed and which assumptions were held fixed. A generated sentence without a corresponding verified model input is not reliable evidence that the stated alternative would change the prediction.

Three practical examples

• A workflow tool shows which adjustable schedule fields would make a proposed plan satisfy its modeled feasibility rule.
• A document classifier demonstrates how removing an ambiguous phrase changes its label, helping analysts inspect behavior.
• A model auditor compares alternative inputs to discover whether predictions depend on implausible or inappropriate feature changes.

Think of it this way

Think of a map showing a nearby point across a boundary: “If your input were here instead, the model's answer would change.” The map can clarify the decision boundary. It does not prove there is a walkable route to that point, that moving there changes the real-world outcome, or that the boundary will remain fixed.

Worked example

Suppose a toy model marks a delivery plan feasible when 2s+r is at least 10, where s is a permitted staffing setting and r is a route-capacity score. For the input s=3,r=2, the score is 8 and the model returns infeasible.
Changing s to 4 while keeping r=2 gives a score of 10 and flips the model output. Alternatively, keeping s=3 and changing r to 4 also reaches 10. Which explanation is more useful depends on feasible actions and costs, not just numerical distance.
If r is a derived value that cannot be changed directly, the second alternative may be unhelpful. If raising staffing also changes route capacity, holding r fixed may be causally unrealistic. The example demonstrates model behavior under explicit inputs, not a verified operational recommendation.

Illustrative case study

Fictional case: Hazel Scheduling adds counterfactual explanations to a planning model. The first version suggests increasing a capacity score without explaining which real action could achieve it. Users cannot act on the advice.
The team distinguishes model explanations from actionable recourse, restricts changes to meaningful controls and tests each proposed alternative against the model. It also shows multiple feasible options when available and records the model version. The revised workflow still requires operational judgment about costs and dependencies. This example illustrates explanation quality; it does not assert that any particular generated alternative will succeed in the real world.

Common misconceptions and mistakes

Small numerical distance is not the same as low effort or feasibility. A counterfactual may change an immutable or sensitive attribute unless constraints prevent it. Correlated features can make independently edited inputs unrealistic.
Do not call a model-level explanation a proven causal effect. A feature change that flips a prediction need not change the actual outcome. Counterfactual explanations can be unstable, nonunique or sensitive to the distance metric. They also do not, by themselves, establish fairness, overall accuracy or compliance with any legal requirement.

Business use

Counterfactual explanations can help users understand individual outputs, identify data errors and inspect decision boundaries. Their value increases when they use familiar units and make permitted changes explicit.
Before presenting them as advice, decide whether the product promises explanation or actionable recourse. Validate alternatives, record assumptions and provide a way to correct inaccurate source data. Evaluate feasibility with people who understand the workflow, and test whether suggestions remain useful across realistic model variation. Avoid presenting one mathematically convenient alternative as the only route forward. Clear limits make the explanation more informative without overstating what the model can establish.

People also ask

Is a counterfactual explanation unique?

No. Several alternative inputs may produce the target output.

Does it prove causation?

Not usually; many methods explain model behavior rather than real-world interventions.

Is it always actionable?

No. Actionability requires additional constraints and evidence about feasible changes.

Check your understanding

A model explanation says that changing a derived score would reverse a decision. Does that automatically tell the user what action to take?

Answer: No. The score may not be directly controllable, and actions affecting it may change other variables. Practical recourse requires a feasible action model and appropriate validation.

Related terms

• Explainable AI
• Decision boundary
• local explanation
• algorithmic recourse
• counterfactual
• feature attribution
• actionability
• causal model

Learn next

Learn the distinction between explaining a prediction and recommending an action. Compare several counterfactual inputs for one toy model, then add feasibility and immutability constraints. Finally, examine whether real-world dependencies make those alternatives achievable.

More in Causal inference and experimentation

• Conditional average treatment effect - The conditional average treatment effect, or CATE, is the average causal effect of a treatment among units with specified characteristics.
• Confounder - A confounder is a variable that, in the relevant causal structure, helps create a noncausal association between an exposure or intervention and an outcome.
• Cooperative inverse reinforcement learning - Cooperative inverse reinforcement learning (CIRL) is a framework in which a human and an artificial agent share an objective, but the agent is uncertain...
• Counterfactual fairness - Counterfactual fairness is a causal-model-based fairness criterion asking whether a prediction for an individual would remain the same in distribution under...
• Cross-validation - Cross-validation evaluates a learning procedure by repeatedly fitting it on one part of a dataset and assessing it on a separate part.
• D-optimal design - D-optimal design selects an experimental allocation that maximizes the determinant of its information matrix.

Sources

Sandra Wachter, Brent Mittelstadt and Chris Russell, Counterfactual Explanations without Opening the Black Box: https://arxiv.org/abs/1711.00399 Author institution overview: https://www.oii.ox.ac.uk/could-counterfactuals-explain-algorithmic-decisions-without-opening-the-black-box/ Primary paper and author-institution excerpts consulted. No legal entitlement claim is made; full technical and editorial review remain pending.

Previous: Counter­balancing Next: Counterfactual fairness