Do both original columns have to be normal for a paired t-test?

The exact small-sample assumption concerns the differences.

Is the sign test a test of the mean?

No; it assesses sign balance under its stated null.

Can I test equivalence with an ordinary difference test?

No; equivalence needs a predefined acceptable margin and a suitable procedure.

AI Glossary / Statistics and probability / Intermediate

Paired difference test

Definition

A paired difference test evaluates a hypothesis using differences between observations that belong together, such as two measurements on the same person or two methods applied to the same item. The paired t-test is a common example, testing a population mean difference. Other paired procedures target signs, ranked differences, or outcome-specific features.

What it means

The key step is to compare within each pair before combining information across pairs. This preserves the connection between measurements and prevents ordinary between-unit variation from being treated as though it were part of the contrast.
The phrase describes a family of approaches rather than one universal test. A test about average change is different from a test about the probability of positive change. Choosing a procedure requires a clear question, an appropriate outcome scale, and assumptions compatible with the study design.

How it works

Define differences consistently, such as d_i = new score minus old score. State the null value, often zero, and whether the alternative is two-sided or directional. Specify that choice before using the observed direction to guide inference.
For n independent pairs, the paired t statistic is (mean difference minus null difference) divided by s_d/sqrt(n), where s_d is the sample standard deviation of the differences. Under independent normal differences, its null distribution is Student t with n minus one degrees of freedom.
The normality assumption for that exact small-sample result concerns the differences, not necessarily the two original measurement columns separately. Large-sample approximations may be useful, but severe outliers, heavy tails, and dependence still matter.
A sign test instead uses whether each nonzero difference is positive or negative. Under a continuous zero-median null with independent pairs, positive signs have probability one-half. This sacrifices magnitude information and answers a different question from a mean-difference test.
The Wilcoxon signed-rank procedure incorporates ranks of absolute differences and requires appropriate symmetry conditions for its usual location interpretation. Ties and zero differences affect its reference distribution and software handling.
A paired randomization or sign-flip test needs a valid assignment or symmetry justification. It is not automatically valid for arbitrary before-and-after data merely because the data are paired.
Report an effect estimate and suitable interval alongside the test result. Missing pairs, repeated pairs from the same unit, and multiple tested outcomes require explicit treatment. A small p-value does not establish practical importance or causal attribution.

Three practical examples

AI evaluation: compare two models' scores on the same independent evaluation items using a clearly defined item-level difference.
Workflow research: evaluate within-person changes while accounting for task order, learning, and other design features.
Method comparison: test a mean difference between two measurement methods, while separately assessing agreement and unacceptable individual discrepancies.

Think of it this way

Think of settling each person's two-sided account before pooling the balances. The test studies the collection of individual balances. Pooling all credits and debits as unrelated observations loses information about which transactions belong together.

Worked example

Suppose six fictional independent evaluation items have paired score improvements of 1, 2, 3, 4, 5, and 6 points. The mean difference is 3.5. The squared deviations from that mean sum to 17.5, giving sample variance 17.5/5 = 3.5.
The standard error is sqrt(3.5/6), approximately 0.7638. For a zero mean-difference null, the paired t statistic is 3.5/0.7638, approximately 4.5826, with 5 degrees of freedom. Its exact t reference requires the stated independent normal-difference model; the sample alone does not verify that assumption.
For comparison, a two-sided sign test observes six positive signs and no negative signs. Under independent equally likely signs with no zero differences, all 2^6 = 64 sign patterns are equally likely. The two most extreme outcomes are all positive and all negative, so the exact two-sided sign-test p-value is 2/64 = 0.03125.
This is not a paired t-test p-value. The sign test uses only direction, while the t statistic uses the mean and variability of magnitudes. Their results should be interpreted according to their respective hypotheses rather than chosen retrospectively for the more attractive conclusion.

Illustrative case study

Illustrative fictional case: Spruce Analytics compares two extraction systems on the same documents. The team prespecifies mean score difference as its main target and retains document-level pairing.
During review, it discovers several documents from each of a few customers. Rather than treating all document differences as independent, it revises the uncertainty calculation to reflect customer clustering. It reports performance differences, failure cases, and the dependence structure together. This fictional case illustrates why pairing alone does not complete the statistical design.

Common misconceptions and mistakes

Do not apply an independent-samples test merely because the two columns have different labels. Do not declare a signed-rank test an assumption-free replacement for a mean test. Avoid switching between one-sided and two-sided alternatives after looking at results.
A nonsignificant difference does not prove equivalence. A significant average difference does not mean every item improved. Also, a before-and-after test does not separate an intervention from concurrent changes without additional design justification.

Business use

Paired difference tests help businesses assess changes on comparable units and quantify evidence about a specified contrast. A useful report states the pairing unit, number of independent pairs, direction and scale of differences, test target, uncertainty, and operational relevance. The result should support a decision alongside reliability, costs, and subgroup behavior.

Check your understanding

Why are there five degrees of freedom in the paired t calculation?

Answer: Six differences are used, and their mean is estimated.

Why is 0.03125 not labeled the t-test p-value?

Answer: It was calculated from the separate sign-test reference distribution.

Related terms

Learn next

Next compare mean, median, and directional targets for paired observations. Then study confidence intervals, practical equivalence margins, and resampling methods that preserve both pairing and higher-level clustering.

Sources

SciPy, scipy.stats.ttest_rel: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html — official paired mean-test implementation. SciPy, scipy.stats.wilcoxon: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html — signed-rank assumptions, ties, and zeros. Both primary documentation pages were read. The t statistic and exact sign enumeration are original calculations.

Previous: Paired data Next: Pairwise metric