Does the method work only for random forests?

No. It can inspect many fitted models as long as their predictions can be scored on altered inputs.

Must the importances add to one?

No. Raw score changes are not generally a normalized allocation, and interactions or correlated inputs complicate such an interpretation.

Why repeat the shuffle?

Different rearrangements produce different changes. Repetition helps show the variability of the estimate for the fixed model and evaluation data.

AI Glossary / Causal inference and experimentation / Intermediate

Permutation variable importances

Definition

Permutation variable importances measure how much a fitted model's performance changes when the values of an input variable are shuffled across evaluation examples. Shuffling disrupts the variable's relationship with the outcome and other inputs while leaving its individual observed values available. A large deterioration suggests that the particular model relies on that variable for the chosen performance measure. The method is also called permutation feature importance.

What it means

Suppose a model predicts delivery delay from distance, parcel weight, and service type. To investigate distance, keep the model fixed but randomly rearrange the distance values among the evaluated parcels. Each parcel now receives a distance that generally belongs to another parcel.

If predictions become noticeably worse, distance was useful to this model under this assessment. If little changes, several explanations are possible: the model may ignore distance, another variable may provide similar information, or the evaluation may be too limited to reveal its contribution.

Importance therefore belongs to a model, dataset, and scoring rule. It is not an intrinsic property of a variable or proof that changing that variable would cause an outcome to change.

How it works

Begin with a fitted model and an evaluation dataset. Measure its baseline score. Choose one variable, shuffle its values among examples, and score the unchanged model on the altered inputs. For a higher-is-better score, importance is the baseline score minus the shuffled score.

Repeat with different random shuffles and summarize the resulting changes. Restore the original data before examining the next variable. When using a lower-is-better loss, describe deterioration as shuffled loss minus baseline loss, or use a scoring convention whose sign is explicit.

Using held-out data helps investigate reliance that contributes to performance beyond the training examples. First establish that the model has useful predictive performance; inspecting a poor model cannot reveal every variable's potential value. Report the evaluation population, metric, repetition count, and variability so the ranking can be interpreted appropriately.

Three practical examples

1. Forecast inspection: analysts shuffle a weather input to assess how strongly an existing demand model relies on it for error reduction on a held-out period.

2. Data-quality prioritization: a service team investigates whether disruptions to particular input fields substantially degrade a deployed candidate's offline predictions.

3. Feature investigation: engineers compare training and held-out permutation results to identify variables that appear useful during fitting but contribute little on new examples.

Think of it this way

Think of rearranging the labels on a set of filing drawers while leaving their contents unchanged. If a clerk's retrieval performance collapses, those labels were useful to that workflow. The analogy is limited because feature shuffling can create combinations that never occur naturally, and other variables may act like duplicate labels. The result reflects reliance under a specific disruption, not a clean causal experiment.

Worked example

Suppose a fictional classifier correctly labels 90 of 100 held-out examples, giving baseline accuracy 0.90. Shuffling the variable “recent service interruptions” three times produces accuracies of 0.76, 0.80, and 0.78.

The score drops are 0.14, 0.10, and 0.12. Their mean is 0.12, or 12 percentage points. This is not a statement that the variable explains 12% of all outcomes or causes a 12% change in customer behavior.

For another variable, three shuffled accuracies are 0.90, 0.91, and 0.89. The corresponding drops are 0, −0.01, and 0.01, averaging zero. On this limited assessment, shuffling that variable shows no consistent deterioration. That does not establish that the variable is universally useless. The small sample, dependence on this fitted model, and random variation all limit the conclusion.

Illustrative case study

Fictional case: Rowan Logistics finds that both “distance in kilometres” and an encoded route-length band have low individual permutation importance. A manager proposes removing both fields because neither appears important.

The analyst notices that the fields contain overlapping information. When one is shuffled, the other can still support the prediction. The team investigates their joint role and evaluates any proposed simplified model after retraining, using appropriate held-out data. It distinguishes this removal experiment from the original fixed-model permutation analysis. The case illustrates correlated-feature interpretation and does not report an actual organization's results.

Common misconceptions and mistakes

Permutation importance is not the same as retraining without a variable. The standard procedure leaves the model fixed, so it measures the effect of disrupting an input the model was trained to expect. Retraining can allow other variables to compensate differently.

Low importance does not prove a variable has no predictive information. Correlated inputs can share information, and a weak model may fail to exploit useful patterns. A negative importance can occur when a shuffle improves the observed score; examine repeated results and possible overfitting rather than assigning a simplistic meaning.

Finally, independent shuffling may create implausible combinations, especially for strongly related variables. Interpret results with knowledge of the data and consider whether a different inspection design is needed.

Business use

The method helps business teams ask focused questions about model reliance, data collection, and monitoring priorities. It can reveal a surprising dependency that deserves investigation or show that an expensive input adds little to the current model's evaluated performance.

Use those findings as starting points for decisions. Before removing a field or changing a workflow, test the actual proposed change. Present score units rather than calling all importance values percentages, and avoid translating a predictive reliance ranking into a list of real-world causes.

Check your understanding

A model's mean absolute error rises from 4 to 7 after a variable is shuffled. What is the deterioration in that loss?

Answer: It increases by 3 units. Because lower error is better, the positive deterioration is shuffled loss minus baseline loss, not the reverse.

Related terms

Data leakage
Random forest
Causal inference
Feature importance
model inspection
correlated features
held-out evaluation
partial dependence

Learn next

Take a small fitted model and record its baseline score. Shuffle one field several times without retraining, then compare the changes. Write a separate sentence describing what the result says about model reliance and what it does not establish about causation.

Sources

Scikit-learn, Permutation feature importance: https://scikit-learn.org/stable/modules/permutation_importance.html

Google, Out-of-bag evaluation, including permutation variable importance: https://developers.google.com/machine-learning/decision-forests/out-of-bag

These primary descriptions support the fixed-model procedure and interpretation limits. Numerical results, case, and exercise are original illustrations, not executed experiments.

Previous: Permutation test Next: Permutational analysis of variance