Must the models use different algorithms?

No. Distinct models within one family can form a Rashomon set.

Does the effect prove a simple accurate model always exists?

No. That depends on the problem and the explored class.

Can two good models disagree on individual cases?

Yes, even when their aggregate scores are very close.

AI Glossary / Statistics and probability / Intermediate

Rashomon effect in machine learning

Definition

The Rashomon effect in machine learning is the presence of multiple different models that predict the same data similarly well. These models may differ in structure, features used, explanations, or predictions for particular cases even when their aggregate performance is close. A Rashomon set formalizes a collection of sufficiently good models using a specified class, evaluation criterion, reference performance, and tolerance.

What it means

A training procedure often returns one model, which can make that model appear uniquely supported by the data. In reality, other models may achieve nearly the same score while telling a different predictive story.
This creates both opportunities and uncertainty. An alternative may be simpler, cheaper to run, or more suitable for operational requirements. At the same time, differences between good models can reveal that an individual prediction or feature-importance claim is less settled than one fitted model suggests. Similar prediction quality does not establish a unique causal explanation.

How it works

Define the candidate model class and the performance criterion. State whether the set concerns empirical loss, a regularized objective, population risk, or another measure. Choose a reference and a tolerance that have a clear interpretation.
For an additive loss tolerance, a set might contain models h satisfying L(h) ≤ L_ref + ε. Other definitions use a relative tolerance, so the rule must be reported. The chosen reference may be a proven optimum or simply the best model found; those are different claims.
Search for alternatives and compare their behavior, not only their aggregate scores. Examine disagreements on individual cases, subgroup performance, complexity, and the features each model relies on. Sampling several models does not usually prove that the entire Rashomon set has been enumerated.
Validate choices using appropriate held-out evidence and operational criteria. Similar training performance can disappear on new data, and selecting repeatedly on a validation set can overfit that set. Model multiplicity should encourage careful comparison rather than unrestricted selection of whichever model supports a preferred conclusion.

Three practical examples

A team finds several near-equal predictors and selects a simpler one that satisfies an operational constraint.
An analyst examines whether a feature remains important across many good models instead of interpreting one model's importance score as definitive.
A decision-support system flags cases where similarly performing models disagree, prompting closer examination of the available evidence.

Think of it this way

Think of several maps that guide travelers through the same town with similar success. One emphasizes major roads, another landmarks, and another footpaths. Their overall usefulness can be similar even when they recommend different routes for a particular journey. Comparing the maps reveals choices hidden by looking at only one.

Worked example

Consider an invented evaluation dataset with 100 binary-labeled cases. Model A makes ten mistakes, model B makes eleven, and model C makes fifteen. Use error rate as loss, take A's 0.10 as the reference, and allow an additive tolerance of 0.02.
The inclusion threshold is 0.10 + 0.02 = 0.12. A and B meet it; C does not. This is a two-percentage-point tolerance, not a two-percent relative increase. A relative tolerance of 2% would instead give 0.10 × 1.02 = 0.102, excluding B.
Suppose A's ten mistakes and B's eleven mistakes overlap on five cases. Their mistake sets have 10 + 11 − 5 = 16 distinct cases. Because the labels and predictions are binary, they disagree exactly where one is wrong and the other is correct: (10 − 5) + (11 − 5) = 11 cases, or 11% of the dataset.
Thus models whose aggregate error differs by only one percentage point can disagree on eleven individual predictions. This small example demonstrates multiplicity and disagreement; it does not show that the models generalize equally well or that either is fair, causal, or operationally preferable.

Illustrative case study

In this fictional case, Sequoia Forecasting compares models for predicting equipment-related support requests. Several candidates have similar validation performance, but one uses fewer inputs and another changes more sharply when a particular feature varies.
The team examines those differences, documents an operational reason for its selection, and checks the chosen model on a reserved evaluation set. It also studies cases on which the candidates disagree rather than treating the winning validation score as the whole decision.
The report identifies the explored candidate family and avoids claiming to have found every near-optimal model. It presents feature-importance conclusions with the multiplicity evidence that supports or weakens them.

Common misconceptions and mistakes

The Rashomon effect does not mean every model is equally valid. Membership depends on the class, data, criterion, and tolerance, and poor models can remain outside the set.
Another mistake is equating similar average performance with identical predictions. The worked example shows how these can diverge. Different parameter values also do not necessarily imply meaningfully different functions; some models have redundant representations.
A Rashomon set is not automatically a Bayesian posterior or a calibrated uncertainty interval. Exploring it can inform uncertainty analysis, but additional assumptions and methods are needed for those interpretations.

Business use

Model multiplicity can give organizations room to choose a model that meets practical requirements without sacrificing much measured predictive performance. It can also expose instability in explanations or decisions that would otherwise be hidden.
Document selection criteria and compare meaningful alternatives. Evaluate how decisions change across the explored models, and avoid presenting a single model's feature relationships as uniquely established by the data. Preserve independent evaluation when using multiple rounds of model comparison.

Check your understanding

Which example models meet the additive threshold?

Answer: A and B.

Why does B fail under a 2% relative tolerance?

Answer: Its 0.11 error exceeds 0.102.

How many individual predictions differ between A and B?

Answer: Eleven, given the specified binary-label mistake overlap.

Related terms

Model selection
Interpretability
Generalization
predictive multiplicity
Rashomon set
feature importance
near-optimal models
model uncertainty

Learn next

Learn next how Rashomon sets are defined and searched for specific model families. Study predictive disagreement and feature-importance ranges, keeping their interpretation distinct from causal identification and probabilistic uncertainty.

Sources

Rudin and colleagues, Position: Amazing Things Come From Having Many Good Models, ICML 2024: https://proceedings.mlr.press/v235/rudin24a.html Xin and colleagues, Exploring the Whole Rashomon Set of Sparse Decision Trees: https://arxiv.org/abs/2209.08040 Machine Learning from Human Preferences, glossary: https://mlhp.stanford.edu/src/glossary.html

Previous: Rasch model Next: Rate - distortion theory