AI Glossary / Probabilistic machine learning / Advanced
Bayesian t-test
Definition
A Bayesian t-test is a Bayesian approach to comparing population means in a setting traditionally analyzed with a t-test. A common version compares a null model of zero standardized mean difference with an alternative model using a Bayes factor, while specifying priors for unknown quantities.
What it means
Instead of reporting how unusual a test statistic would be under a null hypothesis, a Bayes factor compares how well two specified models predict the observed data after averaging over their parameter priors. Evidence can favor either model, or remain inconclusive.
There is more than one Bayesian analysis of a mean difference. This entry focuses on the common Jeffreys - Zellner - Siow, or JZS, Bayes-factor test. Estimating an effect's posterior distribution, testing a point null and assessing whether an effect is practically negligible are related but distinct questions.
How it works
Choose a one-sample, paired or independent-samples formulation appropriate to the design. In a common normal-data JZS model, the alternative places a centered Cauchy prior on standardized effect size δ, while the null fixes δ=0. A compatible shared prior handles the unknown variance.
Compute marginal likelihoods under the two models. BF10 is the alternative's marginal likelihood divided by the null's; BF01 is its reciprocal. Posterior model odds equal prior model odds multiplied by the corresponding Bayes factor. The prior scale for effect size and prior probabilities assigned to models are separate choices. Directional and interval hypotheses require their own explicit formulations.
Three practical examples
- A usability team could compare completion times between two interface designs, after checking that its sampling design and likelihood suit the data. It would distinguish evidence about a mean difference from the size of a useful improvement.
- A training team could analyze paired before-and-after measurements using participants' differences. Treating these as independent groups would ignore the pairing and answer a different statistical problem.
- A laboratory could compare observations with a specified reference mean and examine how conclusions change across plausible effect-size priors. This would make sensitivity visible rather than treating a software default as universally appropriate.
Think of it this way
Imagine two forecasters who publish probability models before seeing a result. Afterward, compare how much predictive support each model assigned to what occurred. A Bayes factor is this relative comparison after accounting for uncertainty within each model. To turn that comparison into updated beliefs about which model is true, you also need the beliefs assigned before seeing the data.
Worked example
Use an original one-sample illustration with ten observations: five equal to −1 and five equal to +1. Their sample mean is zero, their sample variance is 10/9 and the t statistic for a zero population mean is zero. These deliberately simple values support an arithmetic example, not a claim that ten such observations validate a normal model.
Specify a two-sided JZS comparison: H0 fixes δ=0; H1 gives δ a Cauchy prior centered at zero with scale r=1/√2. Use the standard shared Jeffreys variance-prior construction. For n=10 and t=0, evaluating the JZS marginal-likelihood expression gives BF10 approximately 0.308794. Its reciprocal, BF01, is approximately 3.238410.
Thus these data favor H0 over this specified H1 by a factor of about 3.24. That number is neither a p-value nor a probability of 3.24. If the two models receive equal prior probabilities and exhaust the comparison, their posterior odds are 3.238410 to 1 in favor of H0. The corresponding posterior probability is 3.238410/(1+3.238410), approximately 0.764062.
Changing prior model odds changes that posterior probability. Changing the effect-size prior can change the Bayes factor itself. Even this result does not prove exact equality or establish practical equivalence: those interpretations require attention to the models, assumptions and decision question. The numerical integral was independently checked for this stipulated test.
Illustrative case study
Fictional service-design firm Cedar Interface compares two prototypes using task-completion measurements. Before analyzing results, its team defines the comparison, pairing structure and the smallest difference that would matter operationally. It records the effect-size prior used for a Bayes-factor analysis.
The report separates model evidence from estimated effect size and uncertainty. It also examines whether skewed completion times or unusual observations make the chosen likelihood unsuitable, and checks sensitivity to reasonable prior scales. Managers use the analysis alongside implementation costs. This fictional case describes an assessment process; it reports no measured superiority, equivalence or commercial benefit.
Common misconceptions and mistakes
“BF10=0.31 means a 31% probability that the alternative is true.” Posterior probabilities also require prior model odds.
“A Bayes factor favoring the null proves there is no effect.” It expresses relative evidence for the specified models, not certainty.
“A nonsignificant conventional t-test automatically favors the null in a Bayesian test.” The analyses use different quantities; one result does not determine the other without the relevant information and assumptions.
“Default prior scales are identical everywhere.” Software conventions and test formulations can differ; record the numerical scale actually used.
“Evidence for an exact zero answers every equivalence question.” Practical equivalence concerns a substantively defined range, which should be modeled or assessed explicitly.
Business use
Use the method when comparing means is appropriate and stakeholders need a transparent account of evidence between specified models. Document data selection, measurement units, design assumptions, effect-size priors and the direction of the reported Bayes factor. Report effect estimates and their uncertainty alongside the model comparison. Base action thresholds on consequences and practical relevance, rather than treating a descriptive evidence label as an automatic business decision.
People also ask
Can a Bayesian t-test support the null?
Yes, relative to the alternative and priors used in the comparison.
Is the Bayes factor the same as a credible interval?
No. One compares models; the other summarizes a posterior distribution within a specified analysis.
Can the analysis use paired data?
Yes. The model must reflect the pairing, often through within-pair differences.
Check your understanding
What is BF01 when BF10 is approximately 0.308794?
Answer: approximately 3.238410.
What extra assumption gave a posterior null probability near 0.764?
Answer: equal prior probabilities for the two compared models.
Why record the effect-size prior scale?
Answer: it determines the alternative's predictive distribution and can affect the Bayes factor.
Related terms
- Bayes factor
- standardized effect size
- Cauchy prior
- marginal likelihood
- posterior odds
- paired t-test
- credible interval
- practical equivalence
Learn next
Convert Bayes factors and prior odds into posterior odds. Then distinguish a point-null comparison from a posterior interval estimate and an interval-null analysis.
More in Probabilistic machine learning
- Bayesian lasso - The Bayesian lasso is a Bayesian regression approach that uses Laplace, or double-exponential, priors to shrink coefficients toward zero.
- Bayesian logistic regression - Bayesian logistic regression models a binary outcome using a logistic probability function and prior distributions over regression coefficients.
- Bayesian optimization acquisition function - A Bayesian optimization acquisition function scores possible evaluations using the current probabilistic model of an objective.
- Behavior regularization in offline reinforcement learning - Behavior regularization in offline reinforcement learning encourages a learned decision policy to remain close to the actions represented in a fixed...
- Bernoulli mixture model - A Bernoulli mixture model represents binary observations as coming from a mixture of hidden components.
- Beta-binomial model - A beta-binomial model combines a binomial model for a count of successes with a beta distribution for the underlying success probability.
Sources
Morey, R. D. BayesFactor package documentation, ttestBF: https://search.r-project.org/CRAN/refmans/BayesFactor/html/ttestBF.html Faulkenberry, T. J. A tutorial on generalizing the default Bayesian t-test via posterior sampling and encompassing priors, 2019 version, Sections 1–2: https://arxiv.org/pdf/1812.03092 The ten-observation example and fictional case study are original illustrations.
Previous: Bayesian survival analysis Next: Bayesian vector autoregression