Can the null be true exactly?

That depends on the scientific setting and model. Its usefulness also depends on whether testing that claim answers the practical question.

Can a null contain several parameter values?

Yes. Such a null is composite rather than a single fully specified parameter point.

Does rejecting identify the cause of a difference?

No. Causal interpretation requires an appropriate design and additional assumptions.

Is a confidence interval useful alongside the test?

Yes. It helps show which effect magnitudes remain compatible with the analysis and its assumptions.

AI Glossary / Statistics and probability / Beginner

Null hypothesis

Definition

A null hypothesis is a statistical claim used as the reference for a hypothesis test. Often written H0, it specifies a value, relationship, or set of possibilities against which evidence is assessed. It commonly represents no difference or no association, but it can instead specify a nonzero value, a boundary, or an interval. A test evaluates data under the null and accompanying assumptions; it does not automatically assign a probability that the null is true.

What it means

The null hypothesis turns a broad question into a precise statistical comparison. For example, “Does the new interface improve completion?” might become a claim about equality of completion probabilities in a defined population and study period.

The alternative hypothesis specifies the departures of interest. It may allow changes in either direction or only one prespecified direction. Neither hypothesis is simply the researcher's personal belief. Their roles belong to a procedure whose design, error rates, and interpretation need to match the question being asked.

How it works

Identify the population quantity, study design, and null claim before evaluating the results. Select a statistic and determine its distribution under the null model. Use that reference to calculate a p-value or compare the observed statistic with a prespecified rejection threshold.

A small p-value can indicate incompatibility between the observed data and the specified null model, but it does not isolate the null claim from every other assumption used in the calculation. A failure to reject means the procedure did not provide sufficient evidence against the null under its rule; it does not prove exact equality.

Report effect estimates, uncertainty, design limitations, and the relevant practical context. If the objective is to demonstrate practical equivalence or non-inferiority, formulate the hypotheses and margins accordingly. Testing an exact zero difference and obtaining a large p-value is not a substitute for those questions.

Three practical examples

An AI evaluation tests whether two systems have equal mean scores on a prespecified task population.
A quality-control study tests whether a process mean equals a stated target that is not zero.
A survey analysis tests whether two categorical variables are independent under a specified sampling model.

Think of it this way

Think of the null as a reference scenario used to ask how surprising the evidence would be if that scenario and its assumptions applied. It is a starting point for a defined comparison, not a verdict that remains true until one number crosses a line.

Worked example

Consider an original teaching example with ten independent binary outcomes, each having the same success probability p. The null is H0: p = 0.5, and the two-sided alternative is p different from 0.5. Suppose nine successes are observed.

Under the null, the success count K follows a binomial distribution with n = 10 and p = 0.5. For the symmetric exact two-sided test, outcomes at least as far from five successes are K = 0, 1, 9, or 10. Their total probability is [1 + 10 + 10 + 1]/1,024 = 22/1,024, approximately 0.0215.

At a prespecified 0.05 significance level, this procedure rejects the null. The result does not mean there is a 2.15% probability that p equals 0.5, nor that p is certainly above 0.5 in every future setting. With eight successes instead, the corresponding two-sided tail probability would be 112/1,024, approximately 0.1094, so the same procedure would not reject. Independence and a common success probability are explicit teaching assumptions; correlated trials or selective reporting would require a different analysis.

Illustrative case study

Fictional case study: Cedar Compare evaluates a new AI feature against an existing version. The team prespecifies its outcome and a two-sided null of equal population performance. The observed estimate favors the new feature, but the uncertainty interval includes both modest benefit and modest harm.

Management initially calls the versions equivalent because the test does not reject. The analyst explains that the experiment has not ruled out the effects that matter to the decision. Cedar reports the estimate and interval and considers an evaluation designed around a practical equivalence margin. The fictional case shows how clarifying the question can prevent an unsupported conclusion from a non-rejection.

Common misconceptions and mistakes

“The null always says nothing happens.” It can specify a nonzero target or a range of values.

“Failure to reject proves the null.” Limited information may prevent the study from distinguishing meaningful alternatives.

“Rejecting establishes a commercially important effect.” Statistical evidence and practical importance are different questions.

“The p-value is the probability of the null.” A p-value is calculated under the null model, not as a posterior probability for it.

“The null can be rewritten after seeing the outcome.” Data-driven changes can alter the interpretation and error properties of the procedure.

Business use

Businesses should choose hypotheses that reflect the decisions they need to make. A test of any difference may be appropriate for detecting a change, while an adoption decision may depend on whether a benefit exceeds a meaningful threshold or a possible harm is sufficiently small.

Document the analysis plan and distinguish exploratory findings from prespecified comparisons. When many metrics or groups are tested, account for multiplicity and avoid reporting only favorable results. For AI systems, ensure the evaluation population and unit of analysis support the intended claim; a well-calculated test cannot correct a benchmark that answers the wrong operational question.

Check your understanding

A test does not reject equal mean performance. What additional information helps distinguish practical similarity from an inconclusive study?

Answer: The estimated difference, its uncertainty interval, and a prespecified range of practically negligible differences. A p-value alone does not make that distinction.

Related terms

Learn next

Translate a practical question into a precise population claim. Then distinguish one-sided, two-sided, superiority, and equivalence formulations. Practice interpreting a test alongside an effect estimate and interval rather than as a standalone binary conclusion.

Sources

Frank Harrell, Glossary of Statistical Terms, null hypothesis: https://hbiostat.org/glossary/

American Statistical Association, statement release and principles on p-value interpretation: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf

The supplied AIMA glossary interface did not expose its entry. The binomial calculation and Cedar Compare case are original teaching illustrations.

Previous: Null distribution Next: Null move