Is a normal approximation required?

No; the exact calculation uses binomial probabilities.

Is the null always 0.5?

No; specify the probability relevant to the question.

Does non-significance establish equality?

No; it indicates insufficient evidence under the selected test.

AI Glossary / Statistics and probability / Advanced

Binomial test

Definition

A binomial test assesses a specified probability of success using a count of successes in a fixed number of independent binary trials with a common success probability. It compares the observed count with the binomial distribution under the null hypothesis. An exact binomial test uses that discrete distribution directly, without requiring a normal approximation.

What it means

The question is whether the observed count is unusually extreme under a stated probability model. The test does not calculate the probability that the null hypothesis is true. Nor does it establish that a detected difference is operationally important.
Success names whichever outcome is counted. It could mean a correct answer, a selected option or a failed check. Defining that outcome and the alternative hypothesis before examining results prevents an apparently simple test from answering an unintended question.

How it works

Specify the null probability p_0, the fixed trial count n and the observed success count k. Under the null, the probability of j successes is C(n,j) p_0^j (1-p_0)^(n-j).
For an alternative of a larger success probability, add the probabilities for j at least k. For a smaller probability, add those for j at most k. A common two-sided exact convention adds all outcomes whose null probability is no greater than the observed outcome's probability. SciPy implements this probability-ordering convention, with numerical tolerance for floating-point comparisons.
Choose the significance level and test direction before inspecting the outcome. Check independence, a common probability and the sampling plan. Report the observed proportion alongside the test result so readers can assess the size of the difference.

Three practical examples

Simulation validation: an analyst checks whether independent binary outputs are compatible with a configured probability, using a fixed number of runs.
Preference assessment: a researcher examines a predefined yes-or-no choice against a specified baseline, while checking that respondents were sampled appropriately.
Evaluation design: a team tests a predefined success-rate hypothesis on independent cases and separately considers whether the measured difference matters for deployment.

Think of it this way

Imagine a reference machine that repeatedly produces batches under the null probability. The test asks how often that machine would produce a count at least as extreme as the observed one, using the chosen meaning of extreme. It does not reveal the probability that the machine's setting is correct.

Worked example

Consider an original hypothetical experiment with twelve independent trials and ten successes. The null hypothesis is p = 0.5. The observed proportion is 10/12, approximately 0.8333, exceeding the null value by about 0.3333, or 33.33 percentage points.
Under the null, every particular twelve-trial sequence has probability 1/4096. There are C(12,10) = 66 sequences with ten successes, twelve with eleven and one with twelve. For a prespecified greater-than alternative, the p-value is therefore (66 + 12 + 1)/4096 = 79/4096, approximately 0.019287.
For the probability-ordered two-sided test, symmetry adds the equally unlikely counts zero, one and two. Their combined probability is also 79/4096. The two-sided p-value is 158/4096, approximately 0.038574. Both results fall below a prespecified significance level of 0.05, but they answer different alternatives. Choosing the one-sided calculation only after seeing the direction would change the analysis procedure.
Now suppose the same planned experiment had produced nine successes. Its upper-tail probability would include another C(12,9) = 220 sequences, giving 299/4096, approximately 0.072998. The symmetric two-sided result would be 598/4096, approximately 0.145996. One fewer success changes the decision at 0.05, although the estimated success rate, 0.75, still exceeds the null value.
These discrete possibilities explain why an exact test need not have a false-rejection probability exactly equal to its nominal level. In this twelve-trial example, a two-sided 0.05 rule rejects at counts zero through two or ten through twelve. Under the null, those counts occur with probability 158/4096, about 3.86 percent. Exact refers to using the specified distribution, not to attaining exactly five percent.
Doubling a one-sided p-value is not universally the probability-ordered two-sided calculation. For a separate original example, let n = 4, p_0 = 0.2 and k = 2. The probabilities of counts zero through four are 0.4096, 0.4096, 0.1536, 0.0256 and 0.0016. Only counts two, three and four have probability no greater than the observed count's 0.1536. Their sum is 0.1808, the probability-ordered two-sided result. Doubling the upper tail would instead give 0.3616.
All these calculations assume the stated fixed-sample binomial models. Repeatedly checking results and stopping whenever significance appears requires a different assessment of the resulting procedure.

Illustrative case study

Fictional company Alder Testing evaluates a binary simulation output against a predefined baseline. Before running the experiment, its analysts document the outcome definition, trial count, direction of interest and significance threshold. They record both successful and unsuccessful runs.
Their report includes the observed proportion, the null value and the exact test convention. A small p-value prompts investigation of the model and practical consequences; it does not automatically justify changing a product. This fictional scenario reports no measured business improvement.

Common misconceptions and mistakes

A p-value is not the probability that chance caused the result or that the null hypothesis is true. Failure to reject also does not prove equality to the baseline.
Exact computation cannot fix dependent trials, selection bias or a changing success probability. Another mistake is treating a single-proportion binomial test as a general comparison of two independently estimated group proportions. The uncertainty from both groups then needs an appropriate method. Two-sided conventions should be identified rather than assumed interchangeable.

Business use

The test can support transparent checks against a specified binary-outcome benchmark. Decision makers also need the difference's size, uncertainty, sampling context and consequences. Statistical significance alone does not establish that an effect is large enough to justify action.

Check your understanding

Why include eleven and twelve successes in the upper tail?

Answer: They are at least as extreme in the prespecified direction.

Why was the two-sided result doubled in the first example?

Answer: The null distribution was symmetric.

Why was that shortcut wrong in the second?

Answer: Probability ordering selected an asymmetric set of counts.

Related terms

Learn next

Enumerate all possible counts in a small experiment and compare their one-sided and probability-ordered two-sided p-values.

Sources

SciPy, binomtest documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binomtest.html . SciPy implementation, two-sided probability ordering: https://raw.githubusercontent.com/scipy/scipy/v1.17.0/scipy/stats/_binomtest.py . Both numerical experiments and the company scenario are original illustrations.

Previous: Binomial regression Next: BIO tagging