Is the detectable difference the expected effect?

No. It is an effect tied to the study's target power; the expected real effect is a separate assumption or belief.

Can a smaller difference be significant?

Yes. Detectability is probabilistic rather than an absolute cutoff.

Does doubling sample size halve the detectable difference?

Not generally. Under simple independent-data assumptions it often scales approximately with the inverse square root of sample size, so quadrupling is needed to halve it.

AI Glossary / Statistics and probability / Intermediate

Detectable difference

Definition

A detectable difference is a true population effect that would give a specified statistical test a chosen power under stated design and modeling assumptions. It is often called the minimum detectable effect when it marks the effect magnitude achieving the target power. It describes the study's sensitivity, not the smallest effect that can ever produce a significant result and not necessarily the smallest effect worth acting on.

What it means

Power is the probability that the planned test rejects its null hypothesis when a particular alternative is true. For a fixed design, larger differences are often easier to detect than smaller ones. A detectable-difference calculation reverses that relationship: given the sample size, variability, test and desired power, it finds the corresponding effect. A study with a detectable difference larger than the minimum important difference may frequently miss effects that matter. Statistical detectability and practical importance therefore need separate definitions.

How it works

Specify the estimand and effect scale, such as a mean difference or percentage-point difference. Define the test, significance level, one-sided or two-sided alternative, allocation and sample size. Supply assumptions about variation, clustering, pairing, missingness and any planned adjustment. Calculate power across candidate true effects and solve for the effect achieving the target. Some settings have closed-form approximations; others need numerical calculations or simulation. Examine sensitivity to uncertain inputs. The result is conditional on the assumed design and analysis, so changing the outcome, stopping rule or sample structure can change the detectable difference.

Three practical examples

AI experiment planning: determine whether an available user sample can detect an improvement large enough to justify implementation.
Benchmark design: calculate sensitivity for a paired task comparison using assumptions about variability in within-task differences.
Survey research: assess how clustering and unequal allocation affect the differences a planned comparison can detect with adequate power.

Think of it this way

Think of choosing a measuring setup before looking for a signal. Its sensitivity depends on noise and the amount of information collected. A signal smaller than the planned sensitivity target is not impossible to notice, but the chance of noticing it reliably is lower. The value of the signal is a separate question.

Worked example

Consider an idealized comparison of two independent normal populations with known common standard deviation 10 and 100 observations per group. The standard error of the mean difference is 10 sqrt(2/100), approximately 1.4142. For a two-sided z-test at alpha = 0.05, the critical standardized magnitude is 1.95996. Under a positive true difference delta, power is 1 - Phi(1.95996 - delta/1.4142) + Phi(-1.95996 - delta/1.4142). Solving for power 0.80 gives delta approximately 3.9620 outcome units. The familiar approximation using z_0.975 + z_0.80 gives essentially the same result here. Increasing to 400 observations per group halves the standard error and detectable difference under these assumptions. An unknown variance, clustering or a different test requires a different calculation.

Illustrative case study

Fictional case: Pine Product Research can recruit a fixed number of users to compare an assistant workflow. Analysts estimate a plausible range of outcome variability from earlier measurements and calculate detectable differences for the planned test. They find that the available design may have weak power for the smallest improvement stakeholders consider worthwhile. The team considers better measurement, a paired design where justified or more observations, rather than redefining practical importance to match the current sample. This invented case illustrates planning, not a claim about an actual assistant's effect.

Common misconceptions and mistakes

A detectable difference is not a hard boundary between observable and invisible effects. Random variation can produce significance for a smaller true effect or nonsignificance for a larger one. Eighty percent power still allows a twenty percent probability of failing to reject at that specified alternative, under the model. The calculation is not a confidence interval for the observed result. Post hoc power computed from the observed effect adds little to the reported estimate and uncertainty. Multiple outcomes, repeated interim looks and departures from the planned design can invalidate a simple calculation if ignored.

Business use

Businesses should compare study sensitivity with a separately justified minimum worthwhile effect before committing evaluation resources. For AI projects, clarify whether the relevant unit is a user, task, team or model run; counting correlated repetitions as independent can make sensitivity look unrealistically good. Report assumed variability, allocation and analysis alongside the detectable difference. When inputs are uncertain, show a range of scenarios. A nonsignificant study result should be interpreted using its estimate and uncertainty, not dismissed solely because the planning calculation targeted a particular difference.

Check your understanding

Why does the worked example halve the detectable difference only after increasing each group from 100 to 400?

Answer: The standard error scales as one over the square root of group size under the stated model. Quadrupling size halves that standard error, and the effect needed for the same power scales with it.

Related terms

Sample size
Effect size
Significance level
Type II error
Standard error
Sensitivity analysis
Statistical power
minimum detectable effect
minimum important difference
equivalence margin

Learn next

plot power as a function of the true effect for one clearly specified test. Compare sensitivity with a practical decision threshold, then vary the assumed variance and allocation. Study paired and clustered designs before reusing an independent-sample formula for AI evaluations involving repeated or related observations.

Sources

Harrell, Glossary of Statistical Terms, detectable difference: https://hbiostat.org/glossary/ NIST/SEMATECH, Sample sizes required for mean comparisons: https://itl.nist.gov/div898/handbook/prc/section2/prc222.htm NIH Research Methods Resources, Worked Examples for Detectable Difference: https://researchmethodsresources.nih.gov/worked-examples-irgt

Previous: Detailed balance Next: Detection Bias