Does equivalence mean identical?

No. It means sufficiently close under stated bounds and evidence.

Can a result be both significant and equivalent?

Yes, if a small nonzero effect is estimated precisely enough.

Is noninferiority the same?

No. Noninferiority typically concerns one unacceptable direction; two-sided equivalence addresses both bounds.

AI Glossary / Statistics and probability / Intermediate

Equivalence test

Definition

An equivalence test assesses whether an effect lies within prespecified bounds representing differences small enough to be considered practically unimportant for a defined purpose. Its null hypothesis includes effects at or beyond those bounds. Rejecting that null supports practical equivalence under the analysis assumptions. It does not establish exact equality, and a nonsignificant ordinary difference test is not a substitute.

What it means

A team comparing two systems may care whether their average scores differ by more than a tolerable amount, rather than whether their true difference is exactly zero. Equivalence testing makes that tolerance explicit.
For lower and upper bounds L and U, the equivalence alternative is L < delta < U. The null is delta <= L or delta >= U. Bounds can be asymmetric and need not be centered on zero, but they should reflect the actual decision and be justified before inspecting the results.

How it works

Define the estimand and its units, such as a difference in mean response time or a ratio of performance measures. Choose defensible equivalence bounds based on substantive consequences, measurement properties, or an established requirement. Convenient bounds chosen after seeing the estimate undermine the intended evidential interpretation.
A common method is the two one-sided tests procedure, or TOST. One test assesses whether the effect exceeds L; the other assesses whether it is below U. Equivalence is concluded only when both corresponding null hypotheses are rejected at the chosen one-sided significance level alpha.
For standard matched test and confidence-interval constructions, TOST at alpha = 0.05 is equivalent to the entire 90% two-sided confidence interval lying strictly between the bounds. The level is 1 - 2 alpha because the interval combines the two relevant one-sided bounds. No Bonferroni halving is needed for these two components of this single intersection-union test: both must succeed.
Use a method appropriate to the design. Paired observations, unequal group variances, clustered measurements, proportions, and ratios require suitable estimation and uncertainty calculations. The generic logic does not make every normal approximation valid.
Interpret the result alongside ordinary difference testing when useful. A precisely estimated small nonzero effect can be statistically different from zero and still fall within practical equivalence bounds. An imprecise estimate may establish neither a difference nor equivalence. Failure to establish equivalence does not prove a meaningful difference.
Plan precision and sample size for the margins of interest. A narrow tolerance usually requires more information. Multiple endpoints or repeated comparisons may introduce additional error-control questions beyond the two components of one TOST.

Three practical examples

An AI team evaluates whether a compressed model's average quality score stays within a prespecified acceptable difference from a reference model.
An operations team tests whether two measurement workflows have sufficiently similar mean outputs for a defined task.
A researcher assesses whether a correlation is small enough to exclude effects considered practically important.

Think of it this way

Think of checking whether a part fits inside a specified tolerance band. Merely failing to demonstrate that it is oversized does not establish that it fits. An equivalence analysis needs evidence that the relevant quantity stays away from both unacceptable boundaries, allowing for uncertainty.

Worked example

In an original teaching example, define delta as a new process's mean score minus the reference score. Prespecify equivalence bounds of -1 and +1 score unit. Assume the estimated difference has a normal sampling distribution with known standard error 0.3, and observe an estimate of 0.4.
The lower-bound statistic is (0.4 - (-1))/0.3 = 4.6667, with an upper-tail p-value approximately 0.00000153. The upper-bound statistic is (0.4 - 1)/0.3 = -2, with a lower-tail p-value approximately 0.02275. Both are below 0.05, so this TOST supports equivalence within the chosen bounds.
The matching 90% interval is 0.4 plus or minus 1.64485 times 0.3, approximately [-0.0935, 0.8935]. It lies entirely inside (-1, 1).
If the same estimate instead had standard error 0.6, the interval would be approximately [-0.5869, 1.3869], which crosses the upper bound. That less precise result would not establish equivalence. These calculations use a stated known-standard-error normal model; estimated-variance t procedures require their appropriate critical values.

Illustrative case study

Fictional case: Pine Models considers replacing a costly classifier with a smaller one. Before evaluation, the team defines an acceptable average-score difference and plans a paired analysis using the same test items.
The equivalence result supports the average-score margin, but subgroup checks reveal a category with poorer performance. The team reports the precise scope of the result and investigates that category before making a deployment decision. Average equivalence is not presented as proof that every prediction, subgroup, latency measure, or failure mode is equivalent.

Common misconceptions and mistakes

Mistake: interpreting p greater than 0.05 in a difference test as equivalence. Low power can also produce that result.
Mistake: choosing generous margins after observing the data.
Mistake: checking whether only the point estimate is inside the bounds. Uncertainty matters.
Mistake: treating failed equivalence as proof of nonequivalence.
Mistake: confusing equivalence of means with agreement between individual measurements. Substantial individual discrepancies can cancel in the average.
Mistake: treating the confidence interval as a posterior probability statement about a fixed parameter.

Business use

Businesses use equivalence tests when a replacement must be sufficiently similar on a specified measure while offering another benefit, such as lower cost or simpler operation. The tolerance should be connected to real consequences and documented before testing. Report the estimate, interval, bounds, design, and decision rule together. An equivalence result supports the measured claim within its assumptions; it does not automatically approve the whole product or process.

Check your understanding

Why does the first worked interval support equivalence?

Answer: Its entire range lies strictly within the prespecified bounds.

What does the wider second interval establish?

Answer: It does not establish equivalence; further interpretation depends on the full analysis rather than assuming a meaningful difference.

Related terms

Confidence interval
TOST
equivalence margin
smallest effect size of interest
noninferiority
statistical power
intersection-union test
and practical significance

Learn next

distinguish difference, noninferiority, and equivalence hypotheses; justify margins in original measurement units; and plan a paired equivalence study with adequate precision.

Sources

Sources: Lakens, Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses, original article: https://pure.tue.nl/ws/files/80918653/lakeequi2017.pdf . Lakens, Improving Your Statistical Inferences, Equivalence Testing and Interval Hypotheses: https://lakens.github.io/statistical_inferences/09-equivalencetest.html . The normal-model calculation is original.

Previous: Equi-join Next: Error cost function