Is Fisher's test the only exact test?

No. Exact binomial and suitable randomization tests are other examples.

Must an exact test have exactly 5% error at a 5% level?

No. Discreteness can make it conservative.

Does an exact test remove the need for enough data?

No. Small samples can still provide low power and imprecise estimates.

AI Glossary / Statistics and probability / Intermediate

Exact test

Definition

An exact test is a hypothesis test calibrated using a finite-sample null distribution rather than relying on a large-sample approximation. Under its stated assumptions, its rejection probability under the null is controlled at the chosen level, often conservatively when outcomes are discrete. “Exact” describes the statistical calibration; it does not mean the data are error-free or the conclusion is certainly correct.

What it means

A small sample can have a limited set of possible results. An exact procedure calculates how unusual the observed result is within the relevant null distribution. For example, a binomial test can add the probabilities of specified success counts without approximating them by a normal curve.
Exact methods are particularly useful when an approximation is unreliable, but they are not restricted to small datasets. The appropriate method depends on the study design, assumptions, hypothesis, and information conditioned on.

How it works

Specify the null hypothesis, alternative, test statistic, and significance level before examining the result. A one-sided alternative concerns a specified direction. A two-sided alternative concerns departures in either direction, but “at least as extreme” needs a precise definition.
Identify the finite-sample distribution supplied by the model or experimental design. Independent Bernoulli trials with a common success probability give a binomial count. Fisher's exact test for a two-by-two table uses a hypergeometric distribution conditional on the observed margins. A randomization test can instead use the treatment assignments allowed by the experiment.
Calculate probabilities of results as extreme as the observed one under that reference distribution. Complete enumeration is one approach, but formulas and efficient algorithms can avoid listing every outcome. Compare the resulting p-value with the prespecified level.
Discrete probabilities mean that not every nominal significance level is attainable by an ordinary nonrandomized rejection rule. A test set at 5% may have an actual null rejection probability below 5%. This conservatism is compatible with the usual meaning of an exact test.
Distinguish an exact reference distribution from a numerical estimate of its tail probability. A Monte Carlo implementation may estimate an exact-test p-value using random draws; it has simulation variability and needs a valid calculation convention. Merely calling a function with “exact” in its name does not document how the calculation was performed.
Check independence, exchangeability, sampling, and conditioning assumptions. Exact arithmetic cannot repair selection bias, repeated unplanned testing, or a reference distribution inconsistent with the study design. Report the effect estimate and uncertainty as well as the testing result.

Three practical examples

A quality team uses an exact binomial test to compare a success count with a specified reference probability.
A researcher uses Fisher's exact test to analyze a sparse two-by-two table under its conditional sampling framework.
A randomized experiment enumerates permitted treatment assignments to test a sharp null hypothesis using the actual allocation design.

Think of it this way

Think of checking an outcome against a precisely defined catalog of possibilities. The catalog can be exact for the model while the model itself still requires justification.

Worked example

An invented experiment produces seven successes in eight independent trials. Test the prespecified null p = 0.5 against p > 0.5. Under the null, each binary sequence has probability 1/256.
There are eight sequences with seven successes and one with eight. The one-sided p-value is therefore (8 + 1)/256 = 0.03515625. This rejects at a prespecified 5% level.
For a symmetric two-sided binomial test, outcomes with zero, one, seven, or eight successes are at least as extreme. Their combined probability is 18/256 = 0.0703125, which does not reject at 5%.
The observed success proportion is 7/8 = 0.875. Neither p-value is the probability that p equals 0.5, and the one-sided alternative must not be selected after seeing the favorable direction. Independence and the common-probability model are essential to this calculation.

Illustrative case study

Fictional case: Linden Testing reviews a small pilot in which a new classification rule succeeds on seven of eight cases. An analyst initially reports only a favorable one-sided result. The review team checks the original two-sided analysis plan and reports that result instead, alongside the estimated success proportion and the pilot's limited precision. This invented case illustrates that an exact calculation does not excuse changing the question after observing the data.

Common misconceptions and mistakes

Do not interpret “exact” as assumption-free, unbiased, or guaranteed to give the correct decision. Type I and Type II errors remain possible.
A nonsignificant result does not prove equivalence or absence of an effect. A small p-value does not establish practical importance.
Not every two-sided discrete test simply doubles a one-sided p-value. Different definitions of extremeness can produce different valid procedures. State the convention rather than assuming all software must return the same number.

Business use

Exact tests can improve the credibility of analyses with rare events, sparse tables, or small experimental samples. Their usefulness depends on choosing a test that matches the business question and data collection. Teams should document the analysis plan, computational method, and limitations so that a precise-looking result is not mistaken for strong evidence from weak data.

Check your understanding

Why do the worked example's conclusions differ?

Answer: The one-sided and two-sided tests ask different questions.

What makes the calculation finite-sample?

Answer: It uses the eight-trial binomial distribution directly.

Can an exact p-value correct biased recruitment?

Answer: No. That is a data-design problem.

Related terms

Null distribution
Binomial test
Permutation test
Significance level
P-value
Fisher's exact test
discreteness
statistical power

Learn next

Study binomial probabilities and conditional distributions, then compare exact and approximate tests on sparse data. Learn how randomization designs justify their own reference distributions.

Sources

SciPy, binomtest documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binomtest.html SciPy, fisher_exact documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.fisher_exact.html

Previous: Exact match Next: Example-based machine translation