AI Glossary / Natural language and speech / Intermediate

Effect size

Definition

Effect size is a numerical description of how large a difference or association is. In an NLP model comparison, it can be the difference between two systems' scores on a specified metric. It answers “how much?” and should be interpreted alongside uncertainty and practical context, rather than being replaced by a significance-test result.

What it means

If one classifier has 83% accuracy and another has 80%, their observed accuracy difference is three percentage points. That is one possible effect-size measure, expressed on the original metric scale.
Other effect sizes describe relative changes, standardized differences, or associations. These are not interchangeable. A report should state the formula, units, direction, and comparison baseline so the reader knows what the number means.
An observed effect is also an estimate based on particular data and procedures. Its magnitude does not by itself show how precisely it is estimated, whether it will generalize, or whether changing the model caused a business outcome.

How it works

Choose a measure that matches the question and define it before interpreting results. For a simple model comparison, compute both systems' metrics on an appropriate evaluation set and subtract in a stated order.
Report uncertainty using a method suited to the evaluation design. When systems are tested on the same examples, their results are paired; treating them as unrelated samples can misrepresent the uncertainty.
Then assess practical relevance. A small average gain may matter at large scale, while a larger gain may be insufficient if it comes with unacceptable latency, cost, or failures on important cases. Effect magnitude, statistical evidence, and decision value are separate considerations.

Three practical examples

• A classifier report states an absolute accuracy difference in percentage points instead of describing the new model merely as “better.”
• A search evaluation reports the change in a ranking metric together with its uncertainty and the dataset used.
• A product team compares a measured quality improvement with a predefined minimum improvement needed to justify extra operational cost.

Think of it this way

Think of measuring how much two shelves differ in height. The difference tells you the size of the gap, while measurement uncertainty tells you how precisely you know it. The analogy separates magnitude from precision; statistical effects can involve more complex comparisons and may not represent a causal change.

Worked example

Consider invented results from two classifiers evaluated on the same 1,000 examples. Model A gets 800 correct, and model B gets 830 correct. Their accuracies are 80% and 83%.
The absolute difference, B minus A, is 3 percentage points. The relative accuracy increase is (83 − 80) / 80 = 0.0375, or 3.75%. Error rates fall from 20% to 17%, giving a relative error reduction of (20 − 17) / 20 = 15%.
All three numbers describe the same aggregate result using different denominators. Calling the outcome simply “a 15% improvement” would hide which quantity changed.
The counts alone do not tell us which individual examples changed from correct to incorrect or vice versa. That paired information matters for appropriate uncertainty analysis. Nor do these figures establish whether the gain is practically sufficient, how it varies across groups, or whether the next deployment population will show the same difference.

Illustrative case study

Fictional case: EvalBridge reviews a proposed replacement for a ticket-classification model. The presentation highlights a statistically significant improvement but omits the actual difference in the main performance metric. Decision-makers cannot tell whether the gain justifies the slower response time.
The team revises the report to show the observed metric difference, its uncertainty, relevant subgroup results, and the latency tradeoff. It also states the practical improvement threshold agreed for the project.
This fictional case illustrates how effect-size reporting supports a decision. It does not claim that statistical significance is irrelevant or that one numerical threshold is suitable for every application.

Common misconceptions and mistakes

Effect size is not the same as a p-value. A p-value does not directly report the magnitude or business importance of a difference.
Do not confuse percentage points with relative percentage change. Both can be legitimate, but they answer different questions.
Avoid labeling an effect “large” solely by a generic rule without considering the metric and context. Also do not interpret an observational association or model-score difference as proof of a causal business benefit.

Business use

Effect-size reporting makes AI evaluations more useful for launch and investment decisions. It helps stakeholders understand whether a result is substantial enough to matter, while uncertainty shows how firmly that magnitude is established.
A practical report should name the baseline, evaluation population, metric, direction of change, and relevant costs. Where possible, connect the measured difference to a concrete operational interpretation without assuming that every benchmark improvement translates directly into saved time or revenue.

People also ask

Must an effect size be standardized?

No. A raw difference on a meaningful scale, such as percentage points, can be an effect-size measure. Standardization is useful for some questions but changes the interpretation.

Can a statistically significant effect be practically small?

Yes. Statistical evidence and practical magnitude are different, and sample size affects the ability to detect differences.

Does a large observed effect mean the estimate is precise?

No. Magnitude and uncertainty must be reported separately; a large estimate can still be unstable.

Check your understanding

Accuracy rises from 80% to 83%. Is it correct to call this a three-percent relative accuracy increase?

Answer: No. It is three percentage points, while the relative accuracy increase is 3 / 80 = 3.75%. A clear report identifies which calculation it uses.

Related terms

• Statistical significance
• Confidence interval
• Practical significance
• Baseline
• percentage point
• relative change
• paired evaluation
• hypothesis test

Learn next

Recalculate one model comparison as an absolute metric difference and a relative change. Then inspect an uncertainty interval and ask whether the plausible effect sizes meet a stated practical threshold, rather than relying only on a significance label.

More in Natural language and speech

• Dynamic time warping - Dynamic time warping, or DTW, aligns two sequences while allowing differences in their local timing.
• Edge-factored - Edge-factored describes a structured model whose score for a graph or tree decomposes into contributions from its individual edges.
• EDU - EDU stands for elementary discourse unit: a minimal segment used as a building block in a discourse analysis, especially Rhetorical Structure Theory.
• Elman Networks - Elman networks are simple recurrent neural networks that feed the previous hidden state into the computation of the next hidden state.
• Emission probabilities - Emission probabilities describe how likely particular observations are given a hidden state in a probabilistic model such as a hidden Markov model, or HMM.
• Encoder-decoder - An encoder-decoder is a model architecture in which an encoder represents an input and a decoder uses that representation to produce an output.

Sources

Sources: Jurafsky and Martin, Speech and Language Processing, classifier comparison and effect-size discussion: https://web.stanford.edu/~jurafsky/slp3/4.pdf ; NIST/SEMATECH e-Handbook, Quantitative Techniques, practical versus statistical significance: https://itl.nist.gov/div898/handbook/eda/section3/eda35.htm Independently checked the chapter’s metric-difference definition and paired-comparison discussion, and NIST’s distinction between practical and statistical significance.

Previous: EDU Next: Effective branching factor