Must group sizes be equal?

No; the variance formula uses each group's sample size.

Can a contrast compare one group with several others?

Yes, with appropriately chosen coefficients.

Does an interval containing zero prove no difference?

No; it indicates that this procedure has not established a nonzero contrast at the chosen level.

AI Glossary / Statistics and probability / Advanced

Scheffé test

Definition

The Scheffé test is a multiple-comparison procedure for examining contrasts among population means in an analysis-of-variance framework. A contrast is a weighted combination of means whose coefficients sum to zero. Scheffé's method provides simultaneous protection across all such contrasts, including comparisons of groups of means rather than only pairs. Its broad coverage can make it conservative for a narrower question.

What it means

After comparing several groups, an analyst may want to examine more than "group A versus group B." For example, the question might compare one workflow with the average of two alternatives. Both are contrasts.

Testing many selected differences separately at an ordinary significance level can overstate evidence. Scheffé's method addresses an entire family of contrasts among the specified means. That family does not automatically include unrelated outcomes, different fitted models or every analysis in a project.

How it works

For k groups, estimate a contrast C = sum(ci mu_i) by replacing population means with sample means. The coefficients ci must sum to zero. Under the classical independent, normally distributed, common-variance model, estimate the contrast variance as MSE times sum(ci²/ni), where MSE is the pooled within-group mean square.

A simultaneous interval is the estimated contrast plus or minus its standard error multiplied by sqrt[(k - 1)F], using the upper-alpha F critical value with k - 1 and N - k degrees of freedom. An interval excluding zero supports a nonzero contrast at the chosen family level.

This adjustment concerns multiplicity. It does not repair dependence, an inappropriate variance model or a biased sample.

Three practical examples

1. Workflow comparison: compare a new workflow with the equal-weight average of two existing workflows. The coefficients are -1/2, -1/2 and 1.

2. AI evaluation: after comparing several system configurations on an appropriately designed experiment, examine contrasts among their mean scores. Specify whether repeated tasks create dependence that the chosen analysis must represent.

3. Process study: compare groups of operating settings when the questions extend beyond pairwise differences. Document the full set of means defining the contrast family.

Think of it this way

Think of choosing comparisons from a large menu. Scheffé's method accounts for the whole menu of contrasts among the specified means. That breadth costs precision compared with protecting a smaller, suitably chosen collection of questions.

Worked example

Consider fictional summaries for three independently sampled groups A, B and C. Each contains five observations. Their sample means are 10, 12 and 16, and the pooled within-group mean square is four. Suppose the classical normal, common-variance model is appropriate for this teaching example.

There are fifteen observations and three group means, giving twelve residual degrees of freedom. Use a five-percent family significance level. The upper-five-percent critical value from an F distribution with two and twelve degrees of freedom is approximately 3.8853. The Scheffé multiplier is therefore sqrt(2 × 3.8853), approximately 2.7876.

First compare C with the average of A and B. The coefficients (-1/2,-1/2,1) sum to zero, and the estimated contrast is 16 - (10 + 12)/2 = 5.

Its estimated variance is 4[(1/4)/5 + (1/4)/5 + 1/5] = 1.2. The standard error is sqrt(1.2), approximately 1.0954. Multiplying by the Scheffé factor gives a margin of approximately 3.0536.

The simultaneous interval is therefore 5 plus or minus 3.0536, or approximately [1.9464, 8.0536]. It excludes zero, so this contrast is significant within the stated family under the stipulated assumptions.

Now compare B with A. The coefficients (-1,1,0) also sum to zero, and the estimate is two. Its estimated variance is 4(1/5 + 1/5) = 1.6. Applying the same family multiplier gives a margin of approximately 3.5260 and an interval of [-1.5260, 5.5260]. This interval includes zero.

Thus, the procedure can support the first comparison without establishing a difference for the second. Inclusion of zero does not prove A and B are equivalent. Equivalence would require an appropriate question, margin and analysis.

The first comparison uses an equal-weight average of group means. If a business question instead concerns a different mixture of A and B, its contrast coefficients must represent that mixture. The statistical adjustment cannot decide which operational comparison is meaningful.

Illustrative case study

Fictional Cedar Evaluation reviews the worked results. Its initial summary says C is better than both A and B because C exceeds their average.

The reviewer notes that the first contrast concerns an average, not both individual comparisons. Cedar revises the statement to match the coefficients and reports the simultaneous interval. It also avoids describing the nonsignificant A-versus-B result as proof of identical performance.

The team retains the units of the response and considers whether the difference matters operationally. This invented case illustrates interpretation of contrasts; the numbers are not real evidence about a product or intervention.

Common misconceptions and mistakes

"Scheffé covers only pairs." It also covers contrasts comparing weighted groups of means.

"One significant contrast makes every comparison significant." The two worked intervals provide a counterexample.

"The correction validates the sampling assumptions." Multiplicity adjustment does not establish independence or a common variance.

"All possible contrasts means all possible research questions." Coverage is tied to the specified set of means and analysis model.

Business use

Write each intended comparison in plain language and as coefficients. Check that the coefficients sum to zero and that the estimated quantity answers the business question. Explain whether an average weights groups equally or represents an operational mixture.

When only pairwise comparisons matter, consider a procedure designed for that family rather than assuming the broadest family is always preferable. For AI-generated analysis, verify the degrees of freedom, error mean square and interval multiplier. Report effect estimates and uncertainty alongside significance, since an adjusted decision does not measure practical value.

Check your understanding

Why are there twelve residual degrees of freedom?

Answer: Fifteen observations minus three group means leaves twelve.

Why is the first estimated contrast five?

Answer: Sixteen minus the average of ten and twelve equals five.

Why does the second interval not establish equivalence?

Answer: It was constructed to assess a difference, without an equivalence margin.

Related terms

Analysis of variance
Contrast
Familywise error rate
simultaneous confidence interval
Tukey procedure
Bonferroni correction
mean square error

Learn next

Replace the first contrast by C minus a mixture containing three-quarters A and one-quarter B. Recalculate its estimate and standard error while retaining the same Scheffé family multiplier.

Sources

NIST/SEMATECH, Scheffe's method, contrasts and simultaneous confidence intervals: https://www.itl.nist.gov/div898/handbook/prc/section4/prc472.htm

NIST/SEMATECH, Upper Critical Values of the F Distribution: https://www.itl.nist.gov/div898/handbook/eda/section3/eda3673.htm

David C. Howell, Statistical Methods for Psychology glossary, Scheffé test: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html

Previous: Scheduling Next: Scheirer - Ray - Hare test