AI Glossary / AI foundations / Intermediate

Feature attribution

Definition

Feature attribution is a family of explainability methods that assign each input feature a score showing how much it pushed a model's output up or down for a given prediction, or across many predictions. It is not proof of cause and effect, and it describes the model, not the real world. Methods differ in how they compute scores, so two methods can rank the same features differently.

What it means

A model takes many inputs and returns one answer, such as a risk score or a price. Feature attribution breaks that answer into pieces, one per input. A lending model might return a score of 640, and attribution might show that a high card balance lowered the score while a long account history raised it.

There are two common uses. Local attribution explains a single prediction: why was this customer flagged? Global attribution summarizes many predictions: which inputs matter most to the model overall? Business teams use both to check that a model relies on sensible signals, to answer questions from customers and auditors, and to spot inputs that should not be there.

How it works

Most methods start from a baseline, such as an average case or a blank input, and ask how the output changes as each feature moves from the baseline to its actual value. Perturbation methods do this by changing or hiding one feature at a time and re-running the model. Game-theory methods such as Shapley-value approaches try every combination of features and share the change in output fairly among them. Gradient-based methods, used mainly on neural networks, measure how sensitive the output is to small changes in each input.

The result is a signed score per feature. Positive means the feature pushed the output up, negative means it pushed it down, and the scores add up to roughly the gap between this prediction and the baseline. Choice of baseline, treatment of correlated features, and the number of model runs all affect the scores, so settings need to be recorded.

A key limit is that attribution reports what the model used, not what is true. If two features carry the same information, such as income and postal code, credit may be split between them or given to either. Scores can also change when the model is retrained, so they should be re-checked after each update.

Three practical examples

• Finance team: A credit model declines a small-business loan. Local attribution shows that a recent drop in monthly deposits cut the score most, while years in business raised it. The analyst uses this to write a clear reason for the applicant and to check the drop was not a data error.
• Retailer: A demand forecast predicts a sharp rise in sales of umbrellas. Global attribution across the past year shows that rainfall forecast and day of week drive most of the output, while ad spend has almost no weight. The planner stops adjusting ad spend to fix forecasts.
• Support desk: A model routes tickets to a priority queue. Attribution on a sample of tickets shows that the customer's account tier carries more weight than the ticket text. The team reviews whether that is intended or whether it is starving smaller customers of fast replies.

Think of it this way

Think of a restaurant bill split among diners, where each person's share reflects what they ordered and shared. Attribution splits a prediction among inputs in the same way. The analogy stops working because diners are independent, while features often overlap and influence each other, so a fair split is a modeling choice, not a fact.

Worked example

A fictional model scores invoice payment risk from 0 to 100, and the average score across all invoices is 30. For one invoice the model returns 55, which is 25 points above the average. Attribution assigns these contributions: customer paid late twice last year, +20; invoice amount is large, +10; customer has paid on time for 8 years, -7; payment terms are short, +2. Check the arithmetic: 20 + 10 = 30, then 30 - 7 = 23, then 23 + 2 = 25. Baseline 30 plus 25 equals 55, so the pieces add up to the score. The reader learns that late history is the main driver, about 20 of the 25 extra points. If the team removed the late-payment feature entirely, they would expect the score to fall toward 35, but they should re-run the model to confirm, because other features may shift.

Illustrative case study

Fictional case: Northfield Supply, a regional distributor, used a model to flag orders for manual credit review. The team ran attribution on a sample of 500 flagged orders and found that one feature, the sales rep's ID code, carried more weight than payment history. The ID code was acting as a stand-in for which rep handled which region, and some regions had historically been reviewed more strictly. The model had learned the old review habits, not real risk. The team removed the ID code, retrained the model, and re-ran attribution to confirm that payment history and order size now led. They also added attribution checks to each quarterly model review so a similar shortcut would be caught earlier.

Common misconceptions and mistakes

A high attribution score means the feature causes the outcome. It only means the model relied on that feature. Cause and effect needs separate evidence, such as a controlled test.

Attribution gives one correct answer. Different methods and baselines can give different rankings, so state which method and settings were used and test whether conclusions hold across them.

If a feature has a low score, it is irrelevant to the business. Its information may be carried by a correlated feature, or the model may simply not use it. Low weight describes the model, not the world.

Attribution makes a model fair or compliant on its own. It helps you spot problems such as proxy features, but fairness and compliance need testing, policy and human review.

Business use

Start with one model that affects decisions, such as credit, pricing or ticket routing. Pick a standard attribution method, fix the baseline, and run it on a sample of both typical and unusual cases. Ask domain experts to read the results and say whether the leading features make sense. Record the method, settings and findings so that results can be repeated and compared after retraining.

Measure whether attribution leads to action: how many features were removed or fixed, how often reviewers agreed with the explanations, and how stable scores are between model versions. Costs include compute for repeated model runs, analyst time, and the risk of over-trusting a tidy chart. Explanations given to customers should be checked by a person, since a score breakdown can sound more certain than it is.

People also ask

Is feature attribution the same as explainable AI?

No. Explainable AI is the wider goal of making model behavior understandable, and feature attribution is one common technique within it. Others include example-based explanations and simple surrogate models.

Can feature attribution show that a model is biased?

It can reveal warning signs, such as a proxy for a protected trait carrying heavy weight. It cannot prove or rule out bias alone, so it should be paired with outcome testing across groups.

Do I need feature attribution for every model?

Not always. It is most useful where decisions affect people, money or compliance, and where you need to check that the model relies on sensible inputs. For low-stakes uses, lighter checks may be enough.

Check your understanding

In the worked example, why do the contributions add up to 25?

Answer: Because attribution splits the gap between the prediction (55) and the baseline (30). The contributions 20, 10, -7 and 2 sum to 25.

What is the difference between local and global attribution?

Answer: Local attribution explains one prediction. Global attribution summarizes which features matter across many predictions.

Why can a feature with a high score still not be a real cause?

Answer: The score shows what the model used, which may be a stand-in for something else, as the rep ID code was in the case study.

Why should attribution be re-run after retraining?

Answer: Retraining can change how the model uses features, so earlier scores may no longer hold.

Related terms

• Explainability
• Interpretability
• Shapley values
• Feature importance
• Feature engineering
• Model bias
• Black box model
• Model monitoring

Learn next

Pick one decision model in your organization and ask what inputs it uses and whether anyone has checked how much each one matters. Read about explainability and interpretability next, then model bias and model monitoring, to see how attribution fits into wider review.

More in AI foundations

• Expert system - An expert system is a computer program designed to apply specialized knowledge to problems within a defined domain.
• F1 score - The F1 score is a single number between 0 and 1 that summarizes how well a classification model performs by combining precision and recall using their...
• Feature - A feature is one measurable input field a model learns from, such as a customer's age, an order's total or the hour a call arrived.
• Fine-tuning - Fine-tuning is additional training that adapts a pretrained model to a particular task, domain, or desired behavior.
• Floating-point operations (FLOPs) - Floating-point operations (FLOPs) are individual arithmetic steps, such as adding or multiplying two decimal numbers, that a computer performs.
• Frontier model - A frontier model is an AI model at or near the leading edge of capability at a given time, usually a large general-purpose model trained with very large...

Sources

Google, Machine Learning Glossary: https://developers.google.com/machine-learning/glossary NIST, AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework ISO, ISO/IEC 22989 Artificial intelligence concepts and terminology: https://www.iso.org/standard/74296.html The sources were consulted for the general definition of explainability terms and for risk-management context; the examples, analogy, worked example and case are original educational illustrations. Checked 5 Nov 2026.

Previous: Feature Next: Feature cross