Is this the same as a paired t test?

Yes. Matched-samples and paired t test commonly refer to the same basic difference-based test.

Must the two raw samples have equal variances?

No. The ordinary paired test directly models their within-pair differences.

What if one observation in a pair is missing?

A basic complete-pair calculation cannot use that difference. The handling of missingness needs justification rather than automatic deletion without comment.

AI Glossary / Statistics and probability / Intermediate

Matched-samples t test

Definition

A matched-samples t test, also called a paired t test, tests a hypothesis about the population mean of differences between paired numerical observations. Pairs may be repeated measurements on the same unit or observations on meaningfully matched units. The usual null hypothesis is that the mean difference is zero. Computationally, it is a one-sample t test applied to the within-pair differences, rather than an independent-samples comparison of two unrelated groups.

What it means

The key information is how much each pair changes or differs. Large differences between people or tasks can be less distracting when the comparison is made within the same person or task. Whether pairing improves precision depends on how informative that correspondence is.

The test concerns a mean difference, not the claim that every pair has the same outcome or that most pairs improve. A statistically detectable mean change also does not automatically establish an intervention's causal effect. That interpretation depends on how the conditions were assigned and what else changed between measurements.

How it works

For n complete pairs, calculate d_i using a consistent direction, such as new minus old. Compute their mean d-bar and sample standard deviation s_d. To test a hypothesized mean difference δ0, use t = (d-bar − δ0)/(s_d/sqrt(n)), with n − 1 degrees of freedom under the usual model.

An ordinary two-sided confidence interval is d-bar plus or minus the appropriate t critical value multiplied by s_d/sqrt(n). Use the number of pairs, not the total number of individual measurements. The standard deviation required is that of the differences, which generally cannot be recovered from separate standard deviations without information about their covariance.

The exact small-sample reference distribution assumes independent, normally distributed pair differences with common variance. Larger-sample use relies on suitable approximation conditions. Inspect unusual differences, missing pairs, and dependence between pairs. Equal variances of the two original score lists are not a separate requirement of the ordinary paired calculation.

Three practical examples

An evaluation compares two AI systems' numerical scores on the same independently sampled tasks.
A business compares employee completion times under two counterbalanced interface conditions, using a difference for each employee.
A measurement study compares two instruments on the same items to assess a mean difference, while recognizing that mean agreement alone does not establish full interchangeability.

Think of it this way

Think of weighing the contents added to several containers. The containers themselves differ greatly in weight, but comparing each container before and after filling isolates its change. The paired test analyzes those changes. It still needs a credible set of independently sampled containers and an appropriate model for the differences.

Worked example

Consider an original example with five paired score improvements: 1, 2, 3, 4, and 5 points. Their mean is 3. The squared deviations from 3 sum to 4 + 1 + 0 + 1 + 4 = 10, so the sample variance is 10/(5 − 1) = 2.5.

The standard error is sqrt(2.5)/sqrt(5) = sqrt(0.5), approximately 0.7071. Testing a zero mean difference gives t = 3/0.7071, approximately 4.243, with four degrees of freedom. Under the stated normal-difference model, the two-sided p-value is approximately 0.0132.

Using the 95% t critical value of approximately 2.776 gives an interval of 3 ± 2.776 × 0.7071, or about 1.037 to 4.963 points. This interval excludes zero but includes mean improvements below two points.

If the business needs at least a two-point improvement, the zero-difference test does not by itself establish that requirement. The example is deliberately tiny and illustrative; assumptions about the difference distribution cannot be convincingly verified from five observations alone.

Illustrative case study

Fictional case: TaskCedar evaluates an AI drafting aid using paired task scores. Each worker completes comparable tasks with and without the aid, and the order is counterbalanced. The analyst defines the score difference before inspecting results and plots the differences alongside their mean.

Some workers contribute several task pairs, so the team recognizes that an ordinary test treating every pair as independent would understate uncertainty if within-worker dependence is present. It uses an analysis that respects that structure and reports the effect in score units. The case shows why a recognizable paired-test formula does not replace understanding the sampling unit.

Common misconceptions and mistakes

Do not use the standard deviation of one condition in place of the standard deviation of differences. Do not count ten measurements from five pairs as ten independent differences. Sorting the two lists separately destroys the intended pairing even though their individual means remain unchanged.

A small p-value is not the probability that the null hypothesis is true, and a large p-value does not establish equivalence. Normality concerns the differences, not necessarily each original list separately. Switching automatically to a rank test changes assumptions and may change the target question, so alternatives should be chosen deliberately.

Business use

Businesses can use the matched-samples t test to summarize average paired changes when the design, outcome scale, and assumptions support it. Its most useful output is often the mean difference with an uncertainty interval in meaningful units.

Specify the practical threshold before analysis and distinguish statistical detection from business value. Record incomplete pairs and their reasons, because complete-case results may represent a selected subset. For repeated users, tasks, sites, or time periods, examine whether a more structured model is needed. Preserve the original pairing identifiers so the analysis can be reproduced and checked.

Check your understanding

There are 30 complete before-and-after pairs. What degrees of freedom does the ordinary paired t test use?

Answer: It uses 29, because there are 30 pair differences and one estimated mean. The 60 individual measurements are not 60 independent observations for this calculation.

Related terms

Matched samples
Standard error
Degrees of freedom
Confidence interval
paired difference
one-sample t test
repeated measures
equivalence testing

Learn next

Practice calculating a paired test from differences, then study how correlation within pairs affects precision. Learn appropriate methods for clustered pairs, non-normal differences, missing data, and practical equivalence questions.

Missing data

Sources

NIST/SEMATECH, analysis of paired observations: https://www.itl.nist.gov/div898/handbook/prc/section3/prc311.htm

SciPy, official ttest_rel reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html

The five-difference example is original; its statistic, interval, and four-degree-of-freedom tail probability were independently calculated. TaskCedar is fictional.

Previous: Matched samples Next: Matching network