AI Glossary / Statistics and probability / Intermediate

Disordinal interaction

Definition

A disordinal interaction occurs when the direction of a difference between groups reverses across levels of another variable. For example, method A may outperform method B under one condition but underperform it under another. It is often called a crossover interaction because lines representing the group means can cross. The reversal is the defining feature in this glossary's usage; a difference merely becoming larger or smaller without changing sign is not enough.

What it means

The effect of one factor depends on the setting of another, and that dependence changes which group has the higher outcome. An overall average can therefore conceal opposite patterns in subgroups or conditions. This does not make every overall average meaningless: it may answer a clearly specified population-level question. It does mean that the averaging weights and conditional comparisons matter. A visible reversal in sample means is also not automatically evidence that the same reversal exists in the population; uncertainty and study design still require assessment.

How it works

Estimate the relevant group differences separately at each level of the other factor, using a consistent outcome scale and subtraction direction. In a two-by-two setting, compare the two simple differences and their difference-in-differences. A disordinal pattern has opposite signs for the simple differences. Fit an appropriate interaction model and quantify uncertainty in the conditional effects rather than relying only on a plot. With a continuous moderator, examine where the fitted difference crosses zero and whether the observed data support conclusions in that region. Avoid extrapolating a crossing far beyond the measured range or interpreting noisy fitted lines as a demonstrated reversal.

Three practical examples

• AI evaluation: investigate whether one model performs better on short tasks but worse on long tasks under the same scoring definition.
• Product research: compare interface designs separately for novice and experienced users when their relative performance may reverse.
• Operations: assess whether two processing methods change their ordering under low versus high workload conditions.

Think of it this way

Think of comparing two routes in dry and rainy weather. One route may be faster when conditions are dry but slower when it rains. Asking which route is always faster misses the interaction. An average travel time can still be useful if you state how often each type of weather is expected.

Worked example

Suppose hypothetical mean task scores, with higher scores preferred, are 80 for method A and 70 for method B on short tasks. On long tasks they are 60 for A and 75 for B. The A-minus-B difference is +10 on short tasks and -15 on long tasks, so the pattern is disordinal. The change in the difference is -15 - 10 = -25 points. With equal weighting of task types, A averages 70 and B averages 72.5, favoring B by 2.5 points. If the target workload is 80% short tasks and 20% long tasks, A's weighted mean is 76 and B's is 71, favoring A by five points. These specified means establish the arithmetic pattern, not statistical significance or a causal effect. Sample sizes, variability and design would be needed for inference.

Illustrative case study

Fictional case: Fern Evaluation compares two AI summarizers on short and long documents. An aggregate score suggests one winner, but analysts notice opposite rankings across document lengths. They verify consistent scoring and examine uncertainty, task composition and possible leakage before proposing a length-sensitive evaluation strategy. Managers also consider maintenance costs and whether document length is available reliably at decision time. This invented case illustrates how conditional performance can inform further investigation; it does not establish that routing each document to a different model will improve deployed results.

Common misconceptions and mistakes

Do not label every nonparallel plot disordinal: group differences must reverse direction over the relevant conditions. A significant interaction test alone does not establish a reversal, and one significant subgroup effect beside one nonsignificant effect does not establish interaction. Lines drawn between unordered categories are visual aids, not evidence about unobserved intermediate values. A crossover interaction is also distinct from a crossover experimental design in which units receive treatments in different periods. Specify the outcome scale because the interpretation of statistical interaction depends on how effects are represented.

Business use

Businesses can use disordinal patterns to identify where one-size-fits-all decisions may be inappropriate. The next step is to test whether the conditional differences are credible, operationally meaningful and stable in the intended population. Report subgroup estimates and uncertainty, along with any overall result and its weighting scheme. Avoid creating many post hoc subgroups until a reversal appears. A tailored strategy needs its own evaluation because selecting methods using observed subgroup winners can overfit and may introduce implementation costs or new failure modes.

People also ask

Must an interaction cross exactly at an observed factor value?

No. Opposite group orderings at observed levels can imply a crossing between them in a plotted representation.

Does disordinal interaction mean there is no main effect?

No. An average effect can be nonzero, depending on the means and weights.

Can a graph alone prove it?

A graph can show a sample pattern, but inference about the population requires uncertainty assessment and an appropriate analysis.

Check your understanding

Why does changing the workload weights reverse the overall winner in the example?

Answer: A is better on short tasks and B on long tasks. Equal weights favor B slightly, while emphasizing short tasks gives more weight to A's advantage. The conditional means stay fixed; the population-level question represented by the average changes.

Related terms

• Moderation (statistics)
• Simple effect
• Ordinal interaction
• Factorial design
• Subgroup analysis
• Interaction
• moderation
• crossover interaction
• difference-in-differences
• marginal mean
• effect heterogeneity

Learn next

plot cell means for several two-factor examples and distinguish parallel, ordinal and disordinal patterns. Compute simple effects and explicitly weighted averages. Then study confidence intervals for conditional effects and why exploratory subgroup findings need independent confirmation before guiding a tailored policy.

More in Statistics and probability

• Discriminant analysis - Discriminant analysis is a family of statistical methods for distinguishing predefined groups using measured predictors.
• Discrimination - Discrimination, in predictive statistics, is a model's ability to distinguish observations with different outcomes by assigning appropriately ordered...
• Discriminative model - A discriminative model learns to predict an output from an input directly, rather than modeling the full process that generates the inputs.
• Distribution - A distribution describes how values of a variable are spread across possible outcomes or ranges.
• Distributional robustness - Distributional robustness is the ability of a model or decision rule to maintain acceptable performance when the data distribution differs from the one used...
• Distributional similarity - Distributional similarity is similarity inferred from how linguistic items occur across contexts in a body of text.

Sources

David Howell, University of Vermont, statistics glossary: https://www.uvm.edu/~statdhtx/fundamentals9/Glossary/Glossary.html Widaman and colleagues, Distinguishing Ordinal and Disordinal Interactions: https://pmc.ncbi.nlm.nih.gov/articles/PMC3553243/ University of Vermont, Graphing interactions: https://www.uvm.edu/~statdhtx/fundamentals8/SeeingStatisticsApplets/graphint.html

Previous: Disocclusion Next: Disparate impact