Can a null hypothesis imply several possible distributions?

Yes. A composite null can contain unknown nuisance parameters, requiring methods that account for them or use an appropriate invariant or conditional statistic.

Is a permutation distribution always exact?

Exhaustive valid permutations can give an exact conditional reference; randomly sampling permutations gives an approximation with a stated testing convention.

Does failure to reject validate the null model?

No. It may reflect limited information or limited sensitivity to the alternative of interest.

Can the null distribution change with sample size?

Yes. Sample size often affects its spread, shape, or degrees of freedom.

AI Glossary / Statistics and probability / Intermediate

Null distribution

Definition

A null distribution is the probability distribution of a test statistic under a specified null hypothesis and the accompanying assumptions. It provides the reference against which the observed statistic is judged. A null distribution may be derived analytically, approximated asymptotically, generated by a valid randomization or permutation scheme, or estimated through simulation. It is a distribution of possible statistic values, not a probability distribution over whether the null hypothesis is true.

What it means

A test statistic reduces data to a quantity useful for assessing a hypothesis. To decide whether an observed value is unusual, the analyst needs to know what values would arise if the null model applied.

That reference depends on more than the words “no effect.” It can depend on sample size, nuisance parameters, dependence, conditioning, and the sampling or assignment process. Two studies testing superficially similar claims may require different null distributions because their designs or statistics differ.

How it works

Specify the null hypothesis, statistic, and sampling assumptions. Determine the statistic's distribution under those conditions. For some normal-theory tests, a standard t or chi-squared distribution supplies the reference. For a permutation test, valid rearrangements generate statistic values that represent the null under the relevant exchangeability conditions.

Compare the observed value with the appropriate tail or tails. A p-value measures the null probability of results at least as extreme according to the test's stated rule. A rejection threshold can be chosen so that the null probability of entering the rejection region is controlled at a specified level.

Check whether the reference is exact or approximate. A simulation-based estimate also has Monte Carlo uncertainty. Repeated looks, data-dependent model selection, or inappropriate permutations can change the effective reference procedure. The null distribution must describe the analysis actually performed, not an idealized calculation that omits consequential choices.

Three practical examples

An analyst compares an observed t statistic with a t distribution having degrees of freedom determined by the design.
An AI evaluator permutes valid group labels to obtain a reference distribution for a prespecified comparison statistic.
A researcher simulates data from a fully specified null model to investigate the behavior of a complicated statistic.

Think of it this way

Think of a reference collection of scores from a game played under a particular set of rules. An observed score seems unusual only relative to that collection. Changing the rules, number of turns, or scoring method can change which scores count as surprising.

Worked example

Consider an original exact permutation example. Four pooled observations are 1, 2, 4, and 5. Two are assigned to group A and the remaining two to group B. Use the statistic T = mean(A) − mean(B). Under an identical-distribution null with exchangeable labels, each selection of two observations for A has equal probability.

There are six assignments. Choosing A as {1,2}, {1,4}, {1,5}, {2,4}, {2,5}, or {4,5} gives T values −3, −1, 0, 0, 1, and 3, respectively. This is the exact conditional null distribution: −3, −1, 1, and 3 each have probability 1/6, while zero has probability 2/6.

Suppose the observed groups are A = {4,5} and B = {1,2}, giving T = 3. Using absolute magnitude for a symmetric two-sided test, two of six assignments have |T| at least three. The p-value is 2/6, approximately 0.3333. The extreme-looking separation is not rare enough in this tiny experiment to give a small p-value. The calculation depends on the stated exchangeability; freely permuting observations would not automatically be valid for a clustered or paired design.

Illustrative case study

Fictional case study: Cedar Benchmarks compares two AI systems using repeated measurements from the same tasks. An initial script shuffles every recorded score independently, creating a null distribution that ignores which scores belong to the same task.

The analyst reviews the design and replaces the rearrangement scheme with one appropriate to the paired comparison and its assumptions. Cedar documents the statistic and null-generation procedure so another reviewer can reproduce the inference. The fictional case shows that generating many simulated values does not compensate for generating them under the wrong reference structure.

Common misconceptions and mistakes

“The null distribution is the observed data histogram.” It describes the test statistic under specified conditions, not generally the raw observations.

“It gives the probability that the null is true.” It conditions on the null rather than assigning a posterior probability to it.

“Every null distribution is centered at zero.” Its location depends on the statistic; an F or chi-squared statistic has a different reference structure.

“Any shuffle creates a valid null.” Rearrangements must respect the null hypothesis and relevant design assumptions.

Business use

Understanding the null distribution helps businesses assess whether a claimed improvement rests on a defensible comparison. In AI evaluation, task reuse, correlated outputs, filtering, and repeated testing can all affect the reference calculation.

Require enough documentation to identify the analysis unit, statistic, null hypothesis, and reference-generation method. Where simulation is used, retain reproducible settings and report whether the p-value is exact or approximated. A precise-looking result from millions of resamples can still be misleading if the resampling ignores the actual dependence structure or the selection process used to choose the reported finding.

Check your understanding

In the worked example, why does T = 0 have probability 2/6 rather than 1/5?

Answer: Probabilities count equally likely assignments, not merely distinct statistic values. Two of the six assignments produce zero, while the other values occur once each.

Related terms

Learn next

Enumerate a small null distribution by hand and distinguish assignment frequencies from unique statistic values. Then compare an exact reference with a simulation approximation. Study how paired, blocked, and clustered designs constrain valid rearrangements.

Sources

SciPy documentation, permutation_test, null hypotheses, exact enumeration, and randomized-reference conventions: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.permutation_test.html

NIST/SEMATECH e-Handbook, two-sample t-test and reference distributions: https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm

The six-assignment enumeration and Cedar Benchmarks case are original teaching illustrations.

Previous: Nuisance function Next: Null hypothesis