Are variable importance and feature importance the same?

They are commonly used as synonyms, although encoded features may split one original variable into several model inputs.

Should importance scores add to one?

Only if the method normalizes them that way. Other scores use counts, score changes or different units.

Can two methods disagree?

Yes. They may measure different properties, and disagreement can expose redundancy, overfitting or dependence on the evaluation metric.

AI Glossary / Causal inference and experimentation / Intermediate

Variable importances

Definition

Variable importances are scores that describe how input variables matter to a fitted model according to a particular assessment method. Also called feature importances, they can summarize model structure, reliance on information or contributions to predictions. Different methods answer different questions.

What it means

A model may use price, product type and recent demand to predict future sales. An importance chart helps investigate which inputs influence its behavior. The chart does not, by itself, show what causes sales or what would happen if a business changed a price.

Importance belongs to a model, method and evaluation context. A variable with a low score in one model may be useful to a better model, redundant with another input, or important only for a small group.

The plural is useful because there is no single universally correct importance value. A tree's frequency of using a variable and a model's loss of predictive performance when that variable is disrupted provide different information. Their numerical scales need not be comparable.

How it works

Tree-specific methods can count how often a variable appears in split conditions or accumulate improvements attributed to those splits. They describe aspects of the fitted tree structure and training process.

Permutation importance instead assesses a fixed model, shuffles one input column across examples and measures the change in a chosen performance score. Repeating the shuffle helps reveal variation in that estimate. It does not retrain the model after each shuffle.

Interpretation depends on the inputs' relationships. Related variables may provide overlapping information, making individual disruption appear less consequential. Shuffling can also create combinations unlike the original data. A low or high score therefore needs investigation rather than a mechanical conclusion.

First establish that the model predicts usefully on relevant held-out data. Then name the importance method, dataset and performance measure. If using the analysis to select features, assess the resulting selection procedure separately; inspecting a protected test set to redesign the model changes the role of that evidence.

Three practical examples

A forecasting team checks whether a model relies on an unavailable future-data field that accidentally entered its inputs.
A document classifier's developers investigate whether file identifiers rank prominently instead of transferable document characteristics.
An operations team examines whether a costly sensor adds useful predictive information beyond cheaper, correlated measurements.

Think of it this way

Think of examining which clues a person used to solve a puzzle. Counting how often a clue was mentioned differs from observing what happens when that clue is obscured.

Neither exercise shows that the clue caused the underlying event described by the puzzle. Likewise, model importance describes an aspect of prediction, not an automatic causal explanation of the world.

Worked example

A score drop needs a definition

An invented classifier has validation accuracy of 0.90. After shuffling the delivery-distance column several times, its average accuracy is 0.82. Permutation importance under this higher-is-better score is 0.90 - 0.82 = 0.08, an eight-percentage-point drop.

Shuffling package color gives an average accuracy of 0.895, producing a 0.005 drop. These values suggest different levels of reliance for this fitted model on this dataset, subject to the variability across shuffles.

The 0.08 is not an 8% probability that distance caused an outcome. It is also not the performance loss guaranteed if a newly trained model omits distance. Retraining could allow another variable to compensate. The arithmetic measures a specified disruption of the existing model.

Illustrative case study

A fictional delivery company, Fern Dispatch, finds that customer postcode has high structural importance in a tree model. Managers initially propose changing service priorities by postcode on the assumption that it explains late deliveries.

The analysts examine held-out performance and find that postcode is entangled with depot, route and reporting practices. They compare several inspection methods and review individual errors. They also discover that one training field is updated after the delivery outcome is known.

The team removes that leakage and rebuilds the evaluation. It treats the postcode result as a diagnostic lead, not proof of a causal effect or a justification for a service policy. The fictional case illustrates why an appealing ranking requires contextual investigation.

Common misconceptions and mistakes

High importance does not establish causation, direction of effect or permission to intervene on a variable. A global ranking also cannot fully explain every individual prediction.

Tree impurity-based rankings can favor inputs with many possible split values. Prominent placement is therefore not sufficient evidence that an input generalizes well.

Low importance does not prove a variable contains no useful information. Redundancy, poor modeling, the selected metric and the population assessed can all affect the result. Negative permutation importance can occur when shuffling improves the observed score; investigate noise or harmful reliance rather than interpreting it as a universal negative causal effect.

Business use

Use importance analysis to generate questions about data quality, unexpected dependencies and collection costs. Pair rankings with a clear description of the method and the model's held-out performance.

Before removing an expensive input, test the complete revised procedure, including retraining and relevant operating scenarios. Before making a business intervention, seek evidence designed to address causality. Retain traceability from each chart to the model, dataset and metric so a ranking from an earlier version is not mistaken for an enduring fact.

Check your understanding

Accuracy falls from 0.86 to 0.80 after an input is shuffled. Does that show removing the input and retraining must lower accuracy by six percentage points?

Answer: No. The result describes disruption of the fixed model. A retrained model may use remaining information differently.

Related terms

Decision tree
Feature Selection
Data leakage
Causal inference
Feature importance
permutation importance
impurity
correlated features
model inspection

Learn next

Compare one structural importance chart with a held-out permutation analysis for the same model. State what each measures and inspect correlated inputs. Then test a feature-removal proposal through retraining rather than assuming the chart already predicts the outcome.

Sources

Sources: Google, decision forest variable importances: https://developers.google.com/machine-learning/decision-forests/variable-importances Scikit-learn, permutation importance and its limitations: https://scikit-learn.org/stable/modules/permutation_importance.html Google Machine Learning Glossary, variable importances: https://developers.google.com/machine-learning/glossary#variable-importances Scikit-learn, evaluation separation: https://scikit-learn.org/stable/modules/cross_validation.html

Previous: Variable elimination Next: Variable kernel density estimation