Is it restricted to tiny samples?

No, although exact enumeration and optimization can become more computationally demanding.

Is it the same as Fisher's exact test?

No; the conditioning and treatment of possible tables differ.

Can I use it when both models predict the same cases?

Not as an independent-sample comparison. The paired outcome structure must be respected.

AI Glossary / Statistics and probability / Advanced

Barnard exact test

Definition

The Barnard exact test is an unconditional exact procedure for comparing the success probabilities of two independent binomial samples, represented by a two-by-two table. It evaluates possible success counts while keeping the two sample sizes fixed and accounts for the unknown common success probability under the null hypothesis. The ordering statistic and alternative hypothesis must be specified.

What it means

Suppose two independently sampled groups each produce success or failure outcomes. The question is whether their underlying success probabilities differ, rather than merely whether their observed percentages differ.

With small samples or rare outcomes, a large-sample approximation may be unreliable. Barnard's approach evaluates the finite set of possible tables under a binomial sampling model.

Unconditional does not mean that nothing is fixed or assumed. The group sample sizes remain fixed, while the combined number of successes is allowed to vary. This distinguishes the procedure from Fisher's exact test, which conditions additionally on the observed total number of successes. It does not make Fisher's procedure invalid for independent binomial sampling.

How it works

Choose a one-sided or two-sided alternative before examining the result. A common implementation orders tables using a standardized difference between observed proportions. With pooled variance, the statistic divides that difference by the square root of p_hat times (1 - p_hat) times (1/n1 + 1/n2), where p_hat is the combined success proportion.

Enumerate possible success counts from zero to n1 in the first group and zero to n2 in the second. Identify tables at least as extreme as the observed table under the chosen ordering; a two-sided absolute-statistic rule includes both directions.

Under the equality null, both counts follow independent binomial distributions with a common unknown probability pi. Sum the probabilities of the extreme tables for each possible pi, then take the supremum over pi between zero and one. This treatment of the nuisance parameter controls the test's size under the stated model.

Practical implementations perform numerical optimization, so computational settings still matter. Exact refers to the finite-sample statistical construction, not infinite numerical precision. Pooled and unpooled orderings can give different results; report the convention used.

Three practical examples

Product experiment: compare the proportions of independent participants completing a task under two randomly assigned interfaces.
Quality control: compare defect proportions in two independently collected production samples when the counts are small.
Model evaluation: compare success rates from two disjoint independent test samples, while recognizing that two models evaluated on the same cases require a paired analysis instead.

Think of it this way

Think of listing every score two teams could achieve in a fixed number of yes-or-no trials. Mark the scores at least as separated as the observed result, then ask which shared success probability makes that marked region most likely. The test uses that largest null probability.

Worked example

Take two groups of three independent trials each. Group A has three successes and no failures; group B has no successes and three failures. In a table whose columns are the groups and whose rows are success and failure, the entries are [[3, 0], [0, 3]].

The observed proportions are 1 and 0, and the pooled proportion is 0.5. The pooled standard error is sqrt(0.5 × 0.5 × (1/3 + 1/3)) = sqrt(1/6). The standardized difference is therefore sqrt(6), approximately 2.449.

Enumerating all sixteen possible success-count pairs shows that only (3, 0) and (0, 3) have absolute statistics this large. Under a common success probability pi, their total probability is 2 × pi cubed × (1 - pi) cubed.

Because pi × (1 - pi) is maximized at pi = 0.5, the two-sided p-value is 2 × (0.25 cubed) = 0.03125. This independently calculated illustration uses the stated pooled ordering. It does not establish that the observed 100-percentage-point difference is a precise estimate of the population effect.

Illustrative case study

Fictional case: Copper Lane Software pilots two onboarding screens with separate randomly assigned participants. Each participant contributes one binary completion outcome, and the analysis plan specifies an unconditional two-sided comparison.

The analyst checks the table orientation, computes the chosen Barnard test, and reports group counts alongside the estimated difference and an appropriate uncertainty interval. The team also reviews task recordings to understand where participants struggle.

Managers resist treating the pilot's significance decision as a rollout guarantee. They consider recruitment, accessibility, and the range of plausible effect sizes before planning a larger evaluation. The small pilot answers a narrow experimental question, not every question about future customers.

Common misconceptions and mistakes

Do not confuse an exact p-value with the probability that the null hypothesis is true. It describes the extremeness calculation under a null model.

Do not apply the independent-binomial procedure to matched before-and-after outcomes or repeated observations from the same person without addressing dependence.

Do not select a one-sided direction after observing which group performed better. Similarly, comparing several test variants and reporting only the smallest p-value undermines the planned error rate.

Do not assume all procedures called Barnard's test use an identical ordering. Small numerical differences may reflect legitimate conventions or optimization settings that need documentation.

Business use

The test can help teams analyze small binary-outcome experiments without relying solely on an asymptotic approximation. Its usefulness depends on an appropriate sampling design and a decision process that also considers effect size and consequences.

Before use, document success coding, group allocation, sample sizes, exclusion rules, alternative hypothesis, ordering statistic, and software settings. Larger datasets or more complicated designs may call for other approaches. A significance result cannot compensate for biased sampling, outcome redefinition, or unplanned repeated peeking.

Check your understanding

What remains fixed in the standard two-binomial sampling model?

Answer: The two group sample sizes, not necessarily the combined success count.

Why maximize over the common null success probability?

Answer: It is unknown, and the test accounts for the most adverse value for its rejection probability.

Related terms

Binomial distribution
Contingency table
Fisher's exact test
nuisance parameter
hypothesis test
McNemar test
statistical power

Learn next

Compare conditional and unconditional sampling spaces for a small table. Then study paired binary comparisons and confidence intervals for differences in proportions, keeping estimation separate from a significance decision.

Sources

SciPy maintainers, barnard_exact reference documentation, including the sampling model, statistic, nuisance parameter optimization, and implementation settings: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.barnard_exact.html Barnard, G. A., Significance Tests for 2x2 Tables (1947), original publication identified in the official implementation's references: https://doi.org/10.2307/2332517

Previous: Barlow Twins loss Next: Barrier (computer science)