Is non-inferiority the same as equivalence?

No. Equivalence generally requires excluding important differences in both directions; non-inferiority focuses on an unacceptable disadvantage.

Can an intervention be both non-inferior and superior?

Yes, if the evidence supports both claims under an appropriate prespecified testing strategy.

Can the margin be chosen from the observed interval?

No. It needs an independent justification and should be specified before the comparison is evaluated.

AI Glossary / Statistics and probability / Intermediate

Non-inferiority study

Definition

A non-inferiority study is designed to assess whether a new intervention performs no worse than a comparator by more than a prespecified acceptable margin. It does not require the new intervention to be superior or exactly equivalent. The conclusion depends on the outcome, comparison scale, margin, uncertainty interval, and study quality. Although the term is common in clinical research, the same statistical question can arise when evaluating a cheaper, faster, or easier business process.

What it means

Suppose higher scores are better and D is the new intervention's mean minus the comparator's mean. With a positive non-inferiority margin Δ, the study seeks evidence that D is greater than −Δ. The margin marks the largest loss the design is prepared to exclude as unacceptable.

This is different from testing whether D equals zero and obtaining a nonsignificant result. A study with little information can fail to find a difference while still being compatible with a large unacceptable loss. Non-inferiority therefore requires affirmative evidence relative to a justified margin, rather than an absence of evidence for superiority.

How it works

Specify the target population, intervention, comparator, outcome, time horizon, and effect scale. Justify the margin before examining the comparison results, using substantive evidence about acceptable loss rather than choosing a convenient value that reduces sample size.

Plan an analysis and uncertainty interval corresponding to the chosen error rate. For a higher-is-better difference, non-inferiority is supported when the relevant lower confidence bound exceeds −Δ. For lower-is-better outcomes or ratio scales, the direction and boundary must be defined accordingly.

Preserve the study's ability to reveal a meaningful difference. Poor adherence, contamination between conditions, missing outcomes, or insensitive measurement can make interventions look artificially similar. In active-control clinical research, credible interpretation also depends on the comparator retaining an established effect in the new setting. Assess the estimand and sensitivity analyses rather than assuming one analysis population solves every bias concern.

Three practical examples

A company evaluates whether a lower-cost AI model preserves task success within a justified tolerance compared with its current model.
A service team studies whether a shorter training process maintains performance while reducing time away from normal work.
A clinical research team evaluates an intervention against an established active comparator using a carefully supported clinical margin and appropriate trial design.

Think of it this way

Think of replacing a delivery service with a cheaper option while requiring its on-time performance to remain within a specified acceptable loss. Observing similar averages is not enough if the uncertainty still allows a substantial deterioration. The comparison must be precise enough to exclude the loss that matters for the decision.

Worked example

Consider an original simplified AI evaluation with 1,000 independent cases assigned to each model. The new model succeeds on 890 cases and the existing model on 900. Estimated success probabilities are 0.89 and 0.90, giving new minus existing = −0.01, or −1 percentage point.

Using a simple large-sample independent-binomial approximation, the standard error is sqrt((0.89 × 0.11 + 0.90 × 0.10)/1,000), approximately 0.01371. A two-sided 95% interval is −0.01 ± 1.96 × 0.01371, approximately −0.03687 to 0.01687.

If a four-percentage-point margin had been justified and prespecified, the lower bound exceeds −0.04, supporting non-inferiority under this illustrative analysis. It does not support superiority because the interval includes zero.

If the justified margin were three percentage points, the same data would not establish non-inferiority because the lower bound is below −0.03. Researchers cannot choose the wider margin afterward to obtain success. The interval method is a teaching approximation, not a recommended clinical-trial analysis, and paired evaluations would need a different standard error.

Illustrative case study

Fictional case: DraftMaple wants to replace an expensive AI assistant with a smaller system. Before evaluation, the team defines task success and a maximum acceptable loss based on user consequences, then separately measures latency and operating cost.

The quality comparison meets the prespecified non-inferiority rule overall, but a critical task category has too few examples for a reliable conclusion. The team gathers additional evidence for that category instead of interpreting the overall result as a guarantee for every use. The case illustrates why a successful average comparison does not eliminate the need to examine the decision's important dimensions.

Common misconceptions and mistakes

Not significantly different does not mean non-inferior. Non-inferior does not mean identical, superior, or acceptable on every outcome. A result can satisfy a margin even when the new intervention is somewhat worse.

An excessively permissive margin can make a technically successful study unhelpful. Changing the margin after seeing results undermines its meaning. A weak comparator can also make similarity misleading. Errors that dilute real differences may favor a non-inferiority conclusion, so good conduct and appropriate sensitivity analyses are essential rather than optional refinements.

Business use

For businesses, this design can formalize a trade-off when a new option offers a separate benefit, such as lower cost or easier access, while an existing performance standard must be protected. The acceptable loss should reflect the consequences of errors and the people affected.

Report the estimated difference, uncertainty interval, margin, and supporting rationale together. Describe the separate benefit rather than assuming non-inferiority establishes it. Check important subgroups and additional outcomes without overstating underpowered analyses. A deployment decision may require evidence about reliability, usability, or safety beyond the study's primary non-inferiority endpoint.

Check your understanding

A 95% interval for new minus existing performance is −6 to +2 percentage points, with a prespecified margin of three points. Has non-inferiority been established?

Answer: No. The interval still allows losses exceeding three points. Including zero does not turn this inconclusive comparison into evidence of non-inferiority.

Related terms

Confidence interval
Estimand
Non-inferiority margin
superiority
equivalence
assay sensitivity
active comparator
statistical power

Learn next

Study how confidence bounds express one-sided hypotheses, then examine margin justification and the effects of missing data and nonadherence. Distinguish a statistical conclusion from the broader benefit-and-risk decision it informs.

Missing data

Sources

Harrell, glossary, non-inferiority study: https://hbiostat.org/glossary/

FDA, Non-Inferiority Clinical Trials to Establish Effectiveness, methodological guidance: https://www.fda.gov/media/78504/download?attachment=

The AI example and DraftMaple case are original teaching illustrations. The interval arithmetic was independently checked; this entry is an introduction to study design, not clinical or regulatory advice.

Previous: Non-expansive operator Next: Non-linear editing