Does magnitude mean practical importance?

It informs that judgment, but practical importance also depends on context, costs, and consequences.

Is there one universal effect-size measure?

No; different measures answer different questions and use different scales.

Can a nonsignificant estimate still be substantial?

Yes, but its uncertainty may be too large to support a confident conclusion.

AI Glossary / Statistics and probability / Intermediate

Magnitude of effect

Definition

Magnitude of effect describes how large a difference or association is, rather than simply whether it is statistically detectable. In the supplied glossary's analysis-of-variance context, it concerns how much variation in observations is associated with treatments or groups. Measures such as eta squared express this as a proportion of variation. The broader idea also includes differences in original units and other effect-size measures, so a reported magnitude must always identify its scale and definition.

What it means

A significance test and an effect-size estimate answer different questions. A small p-value can accompany a tiny difference when data are abundant. A sizeable estimated difference can remain uncertain when data are limited. Neither result alone tells a business whether a change is worth implementing.

For ANOVA, a variation-based measure summarizes the strength of association between group membership and the response. Saying that variation is “explained” by a model is a statistical description, not automatic evidence that the grouping variable caused the outcome. The study design determines whether a causal interpretation is justified.

How it works

Define the effect of interest and select a measure appropriate to the design. A mean difference preserves the outcome's units; a standardized difference expresses separation relative to variability. In a basic one-way ANOVA, eta squared is the between-group sum of squares divided by the total sum of squares.

Partial eta squared uses the effect sum of squares divided by the sum of that effect and its associated error sum of squares. It can differ from ordinary eta squared in multifactor or repeated-measures designs. State the exact denominator rather than treating the measures as interchangeable.

Report uncertainty and descriptive results. Sample effect sizes can be biased or unstable. Interpret magnitude in context and against a meaningful decision threshold, rather than relying only on generic small, medium, and large labels.

Three practical examples

A training evaluator reports the change in assessment scores and the proportion of variation associated with course groups, alongside uncertainty.
A product experiment compares completion times in seconds so managers can assess whether a statistically detectable improvement matters to users.
An AI evaluation team reports the size of performance differences across model versions rather than presenting only significance-test outcomes.

Think of it this way

Think of measuring both the height of a step and how confidently you have measured it. The height is the magnitude; uncertainty concerns how well you know that height. A precisely measured tiny step and an imprecisely measured large step are different situations, even if a simple significance label makes them seem easy to rank.

Worked example

Consider an original one-way ANOVA illustration with three independent groups of three scores. Group A contains 8, 10, 12; group B contains 10, 12, 14; and group C contains 12, 14, 16. Their means are 10, 12, and 14, and the grand mean is 12.

The between-group sum of squares is 3 times (10 - 12) squared, plus 3 times (12 - 12) squared, plus 3 times (14 - 12) squared = 24. Each group contributes 8 to the within-group sum of squares, giving another 24. Total sum of squares is therefore 48.

Eta squared is 24/48 = 0.5. In this sample and model, half of the total squared variation around the grand mean is associated with differences between group means. The largest pairwise mean difference is 14 - 10 = 4 score units, which provides a complementary practical description.

Eta squared does not indicate which particular comparisons are sufficiently precise, and it does not mean every participant benefits by 50%. If all scores were multiplied by ten, the largest mean difference would become 40 units while eta squared would remain 0.5, because both sums of squares would multiply by 100. The example describes arithmetic on a small invented sample, not a reliable population estimate.

Illustrative case study

Fictional case: Cedar Workshops compares three learning formats. An initial report highlights a statistically significant group comparison but omits the actual score differences. The analyst adds mean estimates, uncertainty intervals, and an explicitly named ANOVA effect-size measure.

Managers discover that the best-performing format also requires considerably more staff time. They compare the plausible score gains with the program's practical goals and costs. The decision uses magnitude and uncertainty together, while the research team separately assesses whether assignment and follow-up support a causal conclusion. A favorable effect-size label is not treated as a complete business case.

Common misconceptions and mistakes

Effect magnitude is not a p-value, and statistical significance does not imply a large or useful effect. A percentage of explained variation is not a percentage improvement in an individual's outcome. Eta squared and partial eta squared may have different denominators and should not be silently substituted.

Avoid comparing standardized measures across studies without examining outcome variability, design, and population. A larger standardized effect can reflect lower background variability rather than a larger raw difference. Also avoid interpreting any estimated sample magnitude as exact; confidence intervals and the study's precision remain important.

Business use

Businesses need effect magnitudes to assess value, plan studies, and compare plausible benefits with implementation costs. Raw-unit effects often make decisions clearer: minutes saved, errors avoided, or changes in completion probability can be directly connected to operational needs.

Variation-based measures are useful complements when summarizing group differences, but they may be less intuitive for decision-makers. Report both when appropriate. In AI projects, identify the evaluation population and metric, examine important subgroups, and avoid equating a large average effect with uniform benefit across every task or user.

Check your understanding

1. What is eta squared in the example?

Answer: 0.5, from 24 divided by 48. 2.

Does that mean a 50% improvement for each person?

Answer: No; it describes a proportion of sample variation. 3.

What happens to eta squared if all scores are multiplied by ten?

Answer: It stays the same, while raw mean differences multiply by ten.

Related terms

Effect size
Eta squared ( h 2 )
Practical significance
Confidence interval
eta squared
partial eta squared
mean difference
standardized effect
ANOVA

Learn next

Compare raw and standardized effects, then study how effect-size definitions change across factorial and repeated-measures designs. Practice interpreting uncertainty and setting a meaningful effect threshold before collecting data.

Effect size

Sources

David Howell, Statistical Methods for Psychology glossary, “Magnitude of effect.” https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html — original ANOVA-oriented scope. GraphPad, “Understanding ANOVA Effect Sizes.” https://www.graphpad.com/guides/prism/latest/statistics/stat_anova_effect_sizes.htm — primary software guidance distinguishing significance, eta squared, partial eta squared, and practical interpretation. The worked sample and fictional case are original.

Previous: Magic set Next: Mahalanobis distance