Does the statistic alone determine a p-value?

Only with the relevant null distribution and definition of extremeness.

AI Glossary / Statistics and probability / Intermediate

Test statistic

Definition

A test statistic is a numerical summary of data used to evaluate a statistical hypothesis. It is chosen to reflect departures relevant to the question and is interpreted using a reference distribution under specified assumptions. The statistic's observed value is distinct from its p-value, critical value, and the practical size of an effect.

What it means

A hypothesis concerns a population or data-generating process, while the available evidence is a finite dataset. A test statistic compresses aspects of that dataset into a quantity that can be compared with what the hypothesized process would tend to produce.
Different statistics detect different kinds of departure. One might focus on a difference in means, another on an association in a contingency table, and another on the largest discrepancy between distributions. No statistic is equally sensitive to every possible alternative.

How it works

State the null hypothesis and the relevant alternative or departure. The null might specify a particular mean, equal group proportions, independence, or a complete probability model. The test statistic should be connected to that question.
Define the calculation before selecting a favorable result. The same observed data can produce many summaries, and choosing whichever appears most unusual changes the testing procedure. Data-driven selection must be included in the uncertainty assessment or evaluated on separate data.
Some statistics standardize an estimated effect by its standard error. For example, a t statistic expresses a mean difference relative to estimated sampling uncertainty. Other statistics are counts, ranks, sums of squared discrepancies, likelihood ratios, or distances and need not have that form.
Determine the null reference distribution. It may follow exactly from a specified sampling model, approximately from an asymptotic result, or through a valid randomization or simulation procedure. Dependence, nuisance parameters, and the study design can change the reference.
Specify what counts as more extreme. A directional alternative may use one tail, whereas a two-sided question must include both relevant directions according to a chosen statistic or ordering. Larger values are not universally the only possible evidence against a null.
Compute the observed value and compare it with critical values or derive a p-value. A p-value concerns the probability of a result at least as extreme under the null and assumptions; it is not the probability that the null is true.
Keep the significance threshold separate from the statistic. A threshold is part of the decision rule. Changing it after seeing the result changes the analysis, and repeated looks at accumulating data require a procedure that accounts for that monitoring.
Report effect estimates and uncertainty in meaningful units as well. A large standardized statistic can result from a small effect measured very precisely. Conversely, a practically important effect may have an inconclusive statistic when data are limited.
Check calculation and interpretation together. Correct arithmetic cannot rescue a test whose sampling assumptions or target hypothesis do not match the data. A failure to reject may reflect low sensitivity rather than confirmation that the null model is correct.

Three practical examples

AI comparison: use a paired mean-loss statistic when both models evaluate the same cases.
Survey analysis: compare observed category counts with counts expected under independence.
Quality checks: evaluate whether a count of successes is unusual under a specified process probability.

Think of it this way

Think of a test statistic as a chosen measuring instrument for disagreement with a hypothesis. The reading has meaning only with a calibration showing what readings the hypothesis would produce. Different instruments respond to different kinds of disagreement.

Worked example

Consider an original toy experiment with six independent binary trials. The null says each trial has success probability one half. Let K be the number of successes and choose T = absolute value of K-3 to detect departures in either direction from the null's expected count.
Suppose K = 5, giving observed T = 2. Under the null, each of the 64 binary sequences is equally likely. Values at least as extreme as T = 2 occur when K is 0, 1, 5, or 6.
There are 1 + 6 + 6 + 1 = 14 such sequences. The exact p-value for this statistic and tail rule is 14/64 = 7/32, or 0.21875. At a preselected 5% threshold, this procedure does not reject the null.
The statistic is 2, while the p-value is 0.21875. They are different quantities. The result does not say the null has probability 21.875%, and it does not prove the process is balanced. With only six trials, the experiment has limited ability to detect many departures.

Illustrative case study

Fictional case: Cedar Experiments displays a test statistic of 2.4 in an AI comparison dashboard without identifying its reference distribution. Managers assume every value above two is significant. Analysts add the statistic definition, degrees of freedom where relevant, p-value, effect estimate, and interval. They also distinguish paired from independent comparisons. The revised report makes clear that a number's interpretation depends on the test and study design, not a universal cutoff.

Common misconceptions and mistakes

Do not confuse the test statistic with the p-value or effect size. There is no universal critical value of two. A statistic's null distribution depends on assumptions and often sample size or degrees of freedom. A nonsignificant test is not proof of no difference. Reporting only the most favorable statistic among many analyses hides part of the testing procedure.

Business use

Use test statistics as one component of a documented evidence process. Preserve their definitions and reference assumptions in analytical reports so results can be reproduced. Keep decisions connected to effect magnitude, uncertainty, costs, and the study's validity rather than reducing them to an unexplained statistic.

Check your understanding

What is the observed statistic in the six-trial example?

Answer: Two.

Which success counts are at least as extreme?

Answer: Zero, one, five, and six.

What is the exact p-value?

Answer: 0.21875, which is not a probability assigned to the null itself.

Related terms

Learn next

Compare t, chi-square, likelihood-ratio, and rank-based statistics. Then examine how a reference distribution is derived and how multiple testing or sequential monitoring changes valid decision rules.

Sources

NIST/SEMATECH, What are statistical tests?: https://www.itl.nist.gov/div898/handbook/prc/section1/prc13.htm Penn State, Hypothesis Testing: https://online.stat.psu.edu/stat555/node/35/ The six-trial enumeration is original.

Previous: Test set Next: Test-time adaptation