Is the sign test the same as a binomial test?

It is a binomial test applied to signs under a one-half null probability.

Does it require normal data?

No, but independence and the relevant sign-probability assumptions still matter.

What happens to zero differences?

A conventional version removes them and reports the reduced sample size; alternative procedures should be explicitly described.

Does a significant result show a large benefit?

No; the signs contain no direct information about the size of differences.

AI Glossary / Statistics and probability / Intermediate

Sign test

Definition

The sign test is a nonparametric test based on whether observations lie above or below a specified value, or whether paired differences are positive or negative. Under its usual null model, the nonzero signs are independent and equally likely to be positive or negative. Their count then follows a binomial distribution with probability one half. For a continuous one-sample distribution, the procedure can test a population median. It does not generally test a mean or measure the magnitude of an effect.

What it means

The test deliberately keeps direction and discards size. A positive difference of one and a positive difference of 100 contribute the same positive sign. This makes it useful when magnitudes are unreliable or only ordered preferences are meaningful, but it also loses information when quantitative differences are informative. For paired data, the natural target is the distribution of within-pair differences. Its median is not generally the difference between the two separate marginal medians.

How it works

Define the comparison and any directional alternative before examining the results. Subtract the reference value from each observation, or calculate each correctly matched pair's difference. Count positive and negative signs. In the conventional implementation, zero differences are excluded and the effective sample size is the number of nonzero differences. Calculate the relevant binomial tail probability under equal sign probabilities. Report ties, direction counts and an effect summary. If ties have substantial probability, explain the conditional nonzero-sign interpretation rather than assuming a continuous-median result automatically applies.

Three practical examples

Example 1: Independent users indicate which of two AI-generated explanations they prefer, with tied preferences recorded separately. Example 2: A researcher tests whether continuous paired task-time differences tend to be positive or negative. Example 3: A quality analyst compares independent measurements with a specified target median. The test can handle ordinal direction information, but observations must still support the independence and sign-probability model. Repeated preferences from the same person cannot simply be counted as unrelated trials.

Think of it this way

Think of a tally sheet with one column for plus and one for minus. Each eligible observation adds one mark, regardless of how large its difference is. The question is whether the imbalance is unusual under an equal-chance sign model. The tally does not tell you whether the positive changes are large enough to matter or whether a few large negative changes outweigh many small positive ones in the mean.

Worked example

Suppose 12 independent paired observations produce nine positive differences, one negative difference and two exact zeros. The conventional sign test excludes the zeros, leaving ten signs. Under the null that a nonzero difference is equally likely to have either sign, the probability of nine or more positives is [ten choose nine + ten choose ten]/2 to the power ten = 11/1,024, approximately 0.010742. The doubled-tail two-sided p-value is 22/1,024 = 0.021484. At a prespecified 0.05 level, the test rejects equal sign probabilities. This conclusion concerns direction among nonzero differences; it does not establish the average size of improvement. Changing one positive difference from nine units to 1,000 units would leave the sign-test result unchanged, although it could greatly change the mean difference.

Illustrative case study

In this fictional case study, Meadow Interface compares two AI explanation formats. Each participant reviews both formats in a balanced order and states a preference, including a tie option. The team uses one preference per participant, reports the number of ties and analyses the non-tied choices with a sign test. It separately examines comprehension scores because preference does not establish improved understanding. The report describes what the sign count supports without claiming that a preferred format delivers a particular numerical productivity gain.

Common misconceptions and mistakes

Common mistakes include treating the test as a general test of means, ignoring pairing, counting repeated observations as independent signs and silently discarding many ties. Another mistake is assuming it requires symmetric quantitative differences; the basic equal-sign test does not require symmetry of magnitudes. The Wilcoxon signed-rank test uses additional information and has different assumptions. A statistically clear directional imbalance can coexist with a small median effect or an unfavorable mean when magnitudes differ substantially.

Business use

Businesses use the sign test for simple paired comparisons, ordered judgments and robust checks of directional change. It is particularly transparent when stakeholders understand counts of favorable, unfavorable and tied outcomes. However, decisions about costs or total benefits usually also need magnitude information. AI evaluations should define the independent unit, such as a participant or task sampled under a specified design, and avoid selecting only favorable comparisons. Present the effect target and tie handling alongside the p-value.

Check your understanding

What is the effective sample size in the worked example?

Answer: Ten, after excluding two zero differences.

What is the two-sided p-value?

Answer: 0.021484 under the stated binomial model.

Would increasing a positive difference's magnitude change this result?

Answer: No, provided its sign stays positive.

Why should ties be reported?

Answer: They affect the denominator and may materially change the interpretation of the comparison.

Related terms

Median
Wilcoxon signed-rank test
Related terms include binomial test
paired comparison
ordinal data
exact test and sign statistic
not just a normality-test result

Learn next

Learn next how exact binomial probabilities are calculated and why discrete tests may have actual significance levels below their nominal targets. Compare mean and median differences using deliberately uneven magnitudes. Then study confidence intervals for a median or sign probability so that the analysis includes an estimate as well as a decision.

Median

Sources

Sources: Penn State STAT 500, Introduction to Nonparametric Tests and Bootstrap, https://online.stat.psu.edu/stat500/Lesson11; NIST Dataplot, Sign Test, https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/signtest.htm. The entry uses the explicit equal-sign probability interpretation and does not generalise the test to arbitrary population means. The binomial calculation and fictional preference study are original illustrations.

Previous: Sigmoid perceptron Next: Signal detection theory