AI Glossary / Statistics and probability / Intermediate
Multinomial test
Definition
A multinomial test evaluates whether observed category counts are compatible with a specified vector of category probabilities under a multinomial sampling model. It extends the idea of testing a binomial probability to more than two categories. The term can refer broadly to multinomial goodness-of-fit procedures or specifically to an exact test. An exact test sums null probabilities over outcomes defined as at least as extreme as the observed counts. The chosen definition of extremeness matters and should be reported, because different valid test statistics can produce different exact p-values.
What it means
The null hypothesis specifies the probabilities of mutually exclusive categories, and the observed counts sum to a fixed total. The question is whether the allocation across categories is unusually discrepant from that null model. The probability of the single observed count vector is not generally the p-value. A p-value includes a defined collection of outcomes that would count as at least as discrepant. It is also not the probability that the null hypothesis is true. Interpretation depends on independent trials, stable category probabilities and a correctly specified sampling design.
How it works
State the null probability vector and verify that it sums to one. Record the observed counts and total sample size. Choose a test statistic or ordering rule, such as Pearson's statistic, the likelihood-ratio statistic or probability-mass ordering. For an exact test, enumerate possible count vectors when feasible and sum their null probabilities whenever their statistic meets the extremeness criterion. For larger problems, use an appropriate computational algorithm or a clearly labeled Monte Carlo approximation. A chi-square approximation may be suitable in some settings, but sparse counts and estimated null parameters require careful treatment.
Three practical examples
- Randomization checks: compare observed allocations with prespecified assignment probabilities under the relevant random mechanism.
- Category monitoring: assess whether a sample's category mix is compatible with a defined baseline probability vector.
- Model evaluation: compare categorical observations with a fully specified probabilistic prediction model when the sampling assumptions apply. If probabilities were estimated from the same observations, the simple fixed-null calculation generally needs modification.
Think of it this way
Think of asking whether a collection of colored tokens looks unusually unbalanced for a stated bag composition. You must decide which possible collections would be considered equally or more surprising, not merely calculate the chance of the exact collection in your hand. The rule for making that comparison is part of the test.
Worked example
Suppose four independent trials have three equally likely categories, and the observed counts are (3,1,0). Under the null, this particular vector has probability [4!/(3!1!0!)] times (1/3)^4 = 4/81. Use probability-mass ordering: include every count vector whose null probability is no greater than 4/81. There are three permutations of (4,0,0), each with probability 1/81, and six permutations of (3,1,0), each with probability 4/81. The exact p-value is therefore 3/81 + 24/81 = 27/81 = 1/3. The remaining vectors have larger individual probabilities and are excluded under this ordering. This original exhaustive example shows why 4/81 is not the p-value. Pearson's statistic for the observed vector is 3.5, using expected counts 4/3 in each category, but the reported p-value here is the exact probability-ordered result, not a chi-square approximation.
Illustrative case study
In a fictional case, Juniper Testing checks whether a small sample of generated labels follows a specified three-category distribution. A reviewer notices that the first report calls the observed vector's probability its significance level. The analyst replaces that calculation with an explicitly defined exact multinomial test and documents the ordering rule. The report also records whether outputs are independent, because repeated labels generated from a shared process state could violate the assumed trial structure. The corrected analysis presents evidence about compatibility, not proof that the generator is perfect.
Common misconceptions and mistakes
A low probability for one particular count vector is not automatically strong evidence against the null. Many individual outcomes can be unlikely even under a correct model. Exact does not mean assumption-free or that the p-value is continuously distributed; discrete tests can be conservative. Different extremeness rules need not agree. Failing to reject does not establish equivalence to the target probabilities. Expected counts, not only the overall sample size, influence the suitability of chi-square approximations. Clustering, changing probabilities and data-dependent category selection can invalidate a naive calculation.
Business use
Businesses should specify the baseline probabilities, sampling unit and test rule before examining the result. Report observed and expected counts, the method used and the practical size of deviations. If simulation estimates the p-value, include enough information about simulation uncertainty and reproducibility. Consider whether the study has adequate power to detect changes that matter operationally. Repeated monitoring requires a plan for repeated testing rather than treating every isolated p-value as an independent alarm. The test is one part of assessing a categorical process, alongside data quality and substantive context.
People also ask
When is an exact test useful?
It is useful when finite-sample probabilities can be computed and an asymptotic approximation is questionable or unnecessary.
Is there only one exact multinomial p-value?
No. The result depends on the specified statistic or ordering of outcomes.
Does a large p-value prove the category probabilities are correct?
No. It indicates that the observed discrepancy is not unusual under the chosen test, not that the null has been proven.
Check your understanding
The observed count vector has null probability 4/81. Must the exact p-value equal 4/81?
Answer: No. It sums probabilities of all outcomes deemed at least as extreme, including the observed vector.
Related terms
- Multinomial distribution
- Goodness-of-fit
- Exact test
- P-value
- Pearson chi-square
- likelihood-ratio statistic
- Monte Carlo test
- discrete inference
Learn next
Enumerate all count vectors for a small multinomial sample and compare probability ordering with other test statistics. Then study how estimated null parameters and repeated testing change the analysis.
More in Statistics and probability
- Multi-class logistic regression - Multi-class logistic regression is a classification method that estimates probabilities for more than two mutually exclusive outcome categories.
- Multinomial Naive Bayes - Multinomial Naive Bayes is a probabilistic classification method commonly used with nonnegative feature counts, especially word counts in text.
- Multinomial probit - Multinomial probit is a statistical model for an outcome that takes one of several unordered categories, often interpreted as a choice among alternatives.
- Multioutput regression - Multioutput regression predicts two or more numerical target variables for each input example.
- Multiple baseline design - A multiple baseline design is a single-case experimental design in which an intervention is introduced at different times across participants, behaviors or...
- Multiple comparison techniques - Multiple comparison techniques are statistical procedures that account for making a family of comparisons rather than interpreting each comparison as if it...
Sources
Resin, A Simple Algorithm for Exact Multinomial Tests: https://arxiv.org/abs/2008.12682 ExactMultinom package authors, documentation of exact tests and alternative statistics: https://stat.ethz.ch/CRAN/web/packages/ExactMultinom/refman/ExactMultinom.html
Previous: Multinomial resampling Next: Multioutput regression