Does 80% power mean an 80% chance the treatment works?

No. It is a rejection probability under a specified assumed effect and design.

Does more data usually increase power?

Under a fixed valid design and effect, more informative data generally help, but dependence and bias must still be considered.

Can power vary across alternatives?

Yes. A power curve describes the rejection probability across different parameter values.

AI Glossary / Statistics and probability / Intermediate

Power

Definition

In statistical hypothesis testing, power is the probability that a specified testing procedure rejects the null hypothesis under a particular alternative scenario. When that alternative makes the null false, power is the probability of detecting the specified departure. It equals one minus the type II error probability for that scenario. Power depends on the test, significance level, sample size, effect size, variability and design. It is not a fixed property of a topic, nor the probability that a statistically significant result is true.

What it means

A study can fail to reject a false null because its data are too variable or too limited to distinguish the chosen effect reliably. Power analysis describes how often the planned procedure would detect an effect if the assumed scenario were repeated. Saying a design has 80% power is incomplete unless the effect, model and decision rule are stated. The same design generally has different power for smaller and larger effects. A study can also be highly powered for a trivial effect while providing little useful information about a meaningful business decision.

How it works

Specify the hypothesis, test statistic, rejection rule and significance level. Choose scientifically or operationally meaningful alternative values and plausible variability, dependence and missing-data assumptions. Calculate the probability that the statistic falls in the rejection region under each alternative, analytically or through simulation. Use these results to plan sample size or assess design trade-offs before collecting data. For clustered, paired, longitudinal or sequential designs, account for their structure rather than using a formula for independent observations. Sensitivity analysis across plausible assumptions is more informative than one calculation based on an optimistic effect size.

Three practical examples

Product experiments: plan the sample needed to detect a minimum meaningful change in a predeclared metric.
Quality research: assess whether an inspection study can detect a specified shift in average performance.
AI evaluation: determine whether a comparison using paired benchmark cases can reliably detect a practically important performance difference under an appropriate model for those paired outcomes.

Think of it this way

Think of a detector with a defined signal size and background noise. Its chance of detecting the signal depends on both the detector's threshold and the strength of the signal relative to the noise. Lowering the threshold may improve detection, but it can also increase false alarms. More reliable measurements can help without changing the decision threshold.

Worked example

Consider an original one-sided test of H0: mu = 0 against mu > 0 using independent normal observations with known population standard deviation 2. Let n = 25 and significance level 0.05. The standard error is 2/sqrt(25) = 0.4, and the standard-normal critical value is approximately 1.6449. Reject when the sample mean exceeds 1.6449 times 0.4, about 0.6580. If the true mean is 1, the standardized cutoff relative to that true mean is (0.6580-1)/0.4, about -0.8551. Power is therefore approximately 0.8038. With n = 16, the standard error is 0.5 and power for the same true mean is approximately 0.6388. These values follow from stipulated known-variance normal assumptions. They are not automatically applicable to unknown-variance, nonnormal, paired or repeatedly monitored studies. Under the null, the planned rejection probability remains 0.05.

Illustrative case study

In a fictional case, Pine Metrics proposes a small experiment because an early pilot showed a large improvement. Analysts calculate power across a range of smaller, still meaningful effects and more cautious variability assumptions. They find that the proposed design is weak for the effect that would actually justify implementation. The team improves measurement consistency and revises the sample plan. The final design record states the target effect and assumptions instead of presenting 80% power as an unexplained quality label.

Common misconceptions and mistakes

Power is not the probability that the null is false, nor the probability that an observed result will replicate exactly. Failure to reject does not establish no effect, especially when the design has low power for relevant alternatives. Raising alpha can increase power but also raises the allowed false-positive rate. Increasing sample size does not fix bias or a poorly defined outcome. Calculating observed power by inserting the observed effect after analysis is generally not a useful substitute for examining the effect estimate and its uncertainty. The analysis should distinguish planned design properties from conclusions about the realized data.

Business use

Businesses should plan around the smallest effect that would change a decision, not merely the most favorable pilot estimate. Include realistic losses from missing data and account for multiple comparisons or interim looks. Improving measurement quality, using pairing where appropriate or balancing groups can sometimes improve efficiency. Report power assumptions transparently so stakeholders understand why a sample size was chosen. After the study, focus on estimated effects, uncertainty and practical consequences, rather than interpreting the original power target as a guarantee that the observed conclusion is correct.

Check your understanding

A study is designed for 80% power at an improvement of five units. Must it have 80% power at one unit?

Answer: No. A smaller effect is generally harder to detect under the same test and sampling conditions.

Related terms

Learn next

Learn to distinguish statistical detection from practical importance, then compare analytic and simulation-based power calculations for the actual dependence and monitoring structure of a study.

Sources

Penn State, Statistical Concepts, Power Analysis: https://online.stat.psu.edu/statprogram/reviews/statistical-concepts/power-analysis Frank Harrell, Glossary of Statistical Terms, power: https://hbiostat.org/glossary/

Previous: Powell's method Next: Power spectrum