AI Glossary / Statistics and probability / Intermediate
T-test
Definition
A t-test is a statistical test that compares a mean or difference in means with a specified null value using an estimated standard error and a Student t reference distribution, either exactly under particular assumptions or approximately. Common forms include one-sample, paired, and independent two-sample tests. The appropriate form depends on the study design.
What it means
An observed difference should be interpreted relative to how variable its estimate could be. A t statistic expresses the difference from the null value in standard-error units. Large magnitudes can indicate incompatibility with the null under the assumptions used to construct the test.
The result does not measure the probability that the null hypothesis is true. Nor does statistical significance establish that a difference is useful, large, or caused by the factor being compared. Report the estimated difference and uncertainty alongside the test.
How it works
Define the target mean or contrast and the null value before looking for a favorable result. Choose whether the scientific question calls for a two-sided alternative or a justified directional alternative. Do not choose the direction after observing the sign.
For a one-sample test, calculate t = (sample mean - null mean)/(sample standard deviation divided by square root of n). Under independent normal observations and a true null, this statistic follows a t distribution with n-1 degrees of freedom.
For paired observations, calculate each pair's difference and apply a one-sample t-test to those differences. Independence is required between pairs, while measurements within each pair are deliberately related. The relevant distributional assumption concerns the differences.
For independent groups with potentially unequal variances, Welch's test uses the square root of s1 squared/n1 plus s2 squared/n2 as the estimated standard error. It uses an approximate degrees-of-freedom calculation rather than assuming a common population variance.
A pooled two-sample t-test assumes equal population variances and uses a pooled estimate. Equal group sizes do not prove that assumption. The choice should reflect the analysis plan and design, not a preference for the smaller p-value.
Compare the statistic with the chosen t reference or calculate the corresponding tail probability. A two-sided p-value includes departures in both directions. State the degrees of freedom and the assumptions behind the calculation.
Assess whether the design and data support the procedure. Strong outliers, heavy tails, small samples, repeated observations, and clustering may undermine a naive calculation. Large-sample robustness is not a license to count dependent rows as independent observations.
Connect the test with an appropriate confidence interval and practical threshold. Failure to reject a zero difference is not evidence of equivalence. Establishing a sufficiently small difference requires a design and analysis that address that specific question.
If many outcomes or comparisons are tested, account for the resulting multiplicity or clearly label exploratory results. A single small p-value selected from many analyses has a different evidential context from a preplanned test.
Three practical examples
- Process checks: compare average measurement output with a specified target.
- Paired AI evaluation: compare average loss differences when two models process the same independent test cases.
- Product experiments: compare mean outcomes between independently assigned groups using an appropriate two-sample method.
Think of it this way
Think of measuring a difference with a ruler whose uncertainty is estimated from the data. The t statistic asks how many estimated uncertainty units separate the observed result from the null value. The reference distribution allows for uncertainty in that ruler itself.
Worked example
Consider an original one-sample example with observations 8, 9, 10, 11, and 12. Test a null population mean of 8 using a preselected two-sided 5% test, assuming independent observations from a normal population.
The sample mean is 10. Squared deviations from 10 sum to 4 + 1 + 0 + 1 + 4 = 10. Dividing by n-1 = 4 gives sample variance 2.5, so the estimated standard error is square root of 2.5/5 = square root of 0.5, approximately 0.7071.
The t statistic is (10-8)/0.7071, approximately 2.828, with four degrees of freedom. The two-sided 5% critical value is approximately 2.776. The observed statistic exceeds it, so this test rejects the null at that level under the stated assumptions.
The corresponding 95% interval is 10 plus or minus 2.776 times 0.7071, approximately 8.037 to 11.963. This tiny dataset is illustrative; the decision does not establish the practical importance or cause of the difference.
Illustrative case study
Fictional case: Juniper Models compares two prediction systems on the same set of customer cases. An analyst initially treats the two columns of errors as independent samples. A review identifies the pairing and calculates within-case loss differences instead. The team reports the mean difference, its interval, and the assumptions about independence across cases. It also checks whether the improvement is large enough to justify deployment costs rather than treating a significance threshold as the whole decision.
Common misconceptions and mistakes
Do not confuse a t-test with standardized assessment T scores. Do not use an independent test for matched observations. A p-value is not the probability that the null is true or that chance caused the result. A nonsignificant result does not prove no effect. With very large samples, small unimportant differences can be statistically detectable. More decimal places do not repair invalid independence assumptions.
Business use
Use t-tests for clearly defined mean comparisons when the design and distributional conditions are suitable. Keep effect estimates and business thresholds central to the report. Document pairing, sample sizes, variance assumptions, and whether the analysis was preplanned so others can interpret the result correctly.
People also ask
Does a t-test require exactly normal data in every practical application?
Exact small-sample reference results use normal assumptions; approximations may be useful under suitable conditions.
Can it compare medians directly?
The usual t-test targets means.
Does significance prove causation?
No.
Check your understanding
What is the worked example's standard error?
Answer: Approximately 0.7071.
What are its degrees of freedom?
Answer: Four.
Why is a paired test appropriate for the same cases evaluated twice?
Answer: It analyzes the within-case differences and respects the pairing.
Related terms
- Standard error
- Confidence interval
- Null hypothesis
- P-value
- Effect size
- Student t distribution
- Welch's test
- paired samples
Learn next
Compare one-sample, paired, Welch, and pooled tests. Then study robustness, equivalence testing, and how clustered observations change standard-error calculations.
More in Statistics and probability
- Synthetic measure - A synthetic measure is a constructed summary that combines several indicators into one value intended to represent a broader concept.
- Synthetic minority oversampling technique - Synthetic minority oversampling technique, commonly called SMOTE, is a training-data resampling method that creates additional minority-class examples by...
- T scores - T scores, in standardized assessment, are scores expressed on a scale with a reference mean of 50 and reference standard deviation of 10.
- Table 2 fallacy - The Table 2 fallacy is the mistake of interpreting every coefficient from one multivariable regression as though each estimated the intended causal effect...
- Taguchi methods - Taguchi methods are approaches to experimental design and quality improvement associated with Genichi Taguchi, emphasizing performance that remains reliable...
- Telephone survey - A telephone survey collects information from respondents through a telephone interview or an automated telephone interaction.
Sources
NIST/SEMATECH, Confidence Limits for the Mean: https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm NIST/SEMATECH, Two-Sample t-Test: https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm NIST/SEMATECH, Student t Critical Values: https://www.itl.nist.gov/div898/handbook/eda/section3/eda3672.htm The five-observation example is original.