AI Glossary / Statistics and probability / Foundation
Two-sided test
Definition
A two-sided test evaluates departures from a null hypothesis in either direction. For a parameter such as a mean difference, a typical alternative is that the difference is not zero, allowing both positive and negative values. It is also called a two-tailed test when the rejection region occupies both tails of a reference distribution. The term describes the alternative hypothesis, not the number of groups or samples in the study.
What it means
A change can matter whether it raises or lowers an outcome. A two-sided test allows either direction to provide evidence against the null while controlling the test's error probability under its assumptions. It therefore differs from a prespecified one-sided test directed at an increase or a decrease. The choice should follow the scientific question and consequences of both directions. Choosing the favorable direction only after seeing the estimate does not preserve the intended one-sided error control.
How it works
Specify the null value and a nondirectional alternative before examining the outcome. Choose a statistic and a definition of at-least-as-extreme results appropriate to its null distribution. For a familiar symmetric normal test of zero, large positive and negative z values count, and the two-sided p-value is twice the smaller one-sided tail probability. That convenient rule is not universal for every discrete or asymmetric test. Use the method's documented two-sided definition, and report the estimate's sign and uncertainty as well as the p-value.
Three practical examples
Example 1: A production study tests whether a process mean has shifted away from a target in either direction. Example 2: An AI experiment tests whether a new interface changes average task performance, recognizing that it could improve or worsen it. Example 3: A fairness analysis examines a prespecified difference between groups without declaring one direction irrelevant. A two-sided test alone does not quantify practical harm or benefit; those interpretations need the observed effect, units, context and uncertainty.
Think of it this way
Think of checking whether a train arrives substantially away from its scheduled time, either early or late. Both departures count as unusual for that question. A different question might focus only on lateness. The test direction expresses which question was asked in advance; it should not be switched after discovering on which side the train happened to arrive.
Worked example
Suppose eight independent Bernoulli trials are observed and the null success probability is one-half. Seven successes occur. Under the null, there are 256 equally likely ordered sequences. For a prespecified upper-sided alternative, the probability of seven or eight successes is (8 + 1)/256 = 0.03515625. For a symmetric two-sided test based on distance from four successes, equally or more extreme counts are zero, one, seven and eight. Their total probability is (1 + 8 + 8 + 1)/256 = 0.0703125. At a 0.05 threshold, these two different tests give different rejection decisions. The upper-sided result is not a valid substitute chosen after observing seven successes. This example has a symmetric null and a stated extremeness rule; it does not establish a universal doubling formula for every two-sided test.
Illustrative case study
In this fictional case study, Hazel Product tests an AI interface change. Its plan uses a two-sided comparison because either improvement or deterioration would matter. The observed estimate is positive, and someone suggests reporting a smaller one-sided p-value. The analyst keeps the planned test, reports the effect and confidence interval, and explains what remains uncertain. Managers assess the practical size and possible downside rather than treating the direction of the observed estimate as permission to change the analysis.
Common misconceptions and mistakes
Common mistakes include confusing two-sided with two-sample, automatically doubling any one-sided p-value, and treating a two-sided rejection as proof of a beneficial change. A rejection indicates incompatibility with the null under the procedure, while the estimate indicates direction. Discrete tests can have several legitimate two-sided conventions that produce different values. Another mistake is assuming that testing in both directions answers equivalence: a nonsignificant two-sided test does not show that the effect is small enough to be practically negligible.
Business use
Businesses can use two-sided tests when both directions are decision-relevant or when a nondirectional question is appropriate. Prespecification helps prevent selective interpretation. Report the actual direction and estimated size, not just whether the test crosses a threshold. If the decision requires demonstrating improvement beyond a meaningful margin, or ruling out unacceptable deterioration, specify that question directly with an appropriate design. Two-sided testing is an inferential choice within a broader decision process, not a substitute for defining operational success.
People also ask
Does a two-sided test require two groups?
No; it can test a single population parameter against a target.
Is its p-value always exactly twice a one-sided value?
No; that depends on the distribution and convention.
Does rejecting the null imply a positive effect?
No; the result may be negative.
Can a one-sided question be legitimate?
Yes, when justified and specified appropriately before the outcome is known.
Check your understanding
Which counts are included in the example's two-sided tail calculation?
Answer: Zero, one, seven and eight.
What is the two-sided p-value?
Answer: 0.0703125.
Why is it larger than the stated upper-sided p-value?
Answer: It includes equally extreme departures in both directions.
What is wrong with selecting the upper-sided test only after seeing seven successes?
Answer: The selection changes the procedure and its error behavior.
Related terms
- Alternative hypothesis
- Rejection region
- P-value
- Significance level
- Related terms include one-sided test
- confidence interval and equivalence testing
Learn next
Learn next how rejection regions are chosen for symmetric and asymmetric distributions. Compare exact discrete two-sided conventions and their interpretation. Then examine the relationship between a compatible two-sided test and confidence interval, while keeping tests of equality distinct from tests designed to establish practical equivalence.
More in Statistics and probability
- Tukey's trend test - Tukey's trend test is an approach to detecting a dose-response trend by examining regression slopes under several prespecified dose scales, commonly...
- Two-proportion Z-test - A two-proportion Z-test is a large-sample hypothesis test comparing two independent population proportions.
- Two-sample hypothesis testing - Two-sample hypothesis testing uses observations from two samples to assess a specified claim about their source populations or distributions.
- Twoway factorial design - A two-way factorial design studies two factors by including every combination of their selected levels.
- Type I error - A Type I error occurs when a statistical hypothesis test rejects a null hypothesis that is true.
- Type II error - A Type II error occurs when a statistical test fails to reject a null hypothesis that is false.
Sources
Sources: Harrell, Glossary of Statistical Terms, https://hbiostat.org/glossary/; NIST/SEMATECH, Critical values and p values, https://itl.nist.gov/div898/handbook/prc/section1/prc131.htm; Relationship between a test and a confidence interval, https://itl.nist.gov/div898/handbook/prc/section1/prc15.htm. These primary author and institutional sources explain directional alternatives and reference inference. The eight-trial calculation is original and independently checked; the product case is fictional.
Previous: Two-scale style transfer Next: Two-stream self-attention