Is KS only a test of equal means?

No. It compares cumulative distributions and can respond to other differences.

Does a large p-value prove the null distribution?

No. It indicates insufficient evidence against that null under the test and its assumptions.

Can I estimate parameters first?

You can, but the null calibration must account for that estimation rather than automatically using the ordinary fully specified test.

AI Glossary / Statistics and probability / Intermediate

Kolmogorov-Smirnov test

Definition

The Kolmogorov - Smirnov test is a distribution-comparison test based on the largest vertical difference between cumulative distribution functions. The one-sample form compares an empirical distribution with a specified continuous reference distribution. The two-sample form compares the empirical distributions of two independent samples. It can detect differences in location, spread or shape, rather than only a difference in means. Its familiar null calibration depends on assumptions, including continuity in the standard form, and must be reconsidered when parameters are estimated from the same data or observations are dependent.

What it means

A cumulative distribution function, or CDF, gives the fraction of probability at or below each threshold. The empirical CDF estimates that fraction using observed data and rises in steps. The KS statistic looks for the largest gap between the curves anywhere along the measurement scale. This provides an overall measure of discrepancy, but it does not explain why the distributions differ or whether the difference matters operationally. A large sample can make a small discrepancy statistically detectable, while a small sample can miss differences that would matter in practice.

How it works

For a one-sample test, sort observations x₍₁₎ through x₍ₙ₎ and specify the null CDF F₀ completely. Compute D+ = max[i/n − F₀(x₍ᵢ₎)] and D− = max[F₀(x₍ᵢ₎) − (i − 1)/n]. The two-sided statistic is D = max(D+, D−). Both sides of each empirical jump matter. In a two-sample test, use the maximum absolute difference between the two empirical CDFs. Convert the statistic to a p-value using an appropriate exact, asymptotic or resampling calibration for the situation. A one-sided alternative concerns the direction of the CDF difference, which should be distinguished from the direction of measurement values.

Three practical examples

Distribution checking: compare measurements with a fully specified continuous reference model when the sampling assumptions are defensible.
Data monitoring: compare a continuous feature's distribution across two independently sampled periods, while accounting for repeated testing and time dependence where necessary.
Simulation validation: inspect whether generated values match a target distribution, using plots and additional checks alongside a test. A nonsignificant result alone does not certify the generator.

Think of it this way

Think of two stairways or curves showing cumulative progress through the data. KS measures their widest vertical separation. It does not average all separations and does not focus specifically on the most extreme tail observations. The location of the widest gap can help an analyst investigate the result, but the test statistic itself only summarizes the largest discrepancy under the selected comparison.

Worked example

Suppose four independent observations are 0.10, 0.40, 0.45 and 0.90, and the null is a Uniform(0, 1) distribution specified in advance. Since F₀(x) = x in this interval, the D+ candidates are 0.25 − 0.10 = 0.15, 0.50 − 0.40 = 0.10, 0.75 − 0.45 = 0.30 and 1.00 − 0.90 = 0.10. The D− candidates are 0.10 − 0 = 0.10, 0.40 − 0.25 = 0.15, 0.45 − 0.50 = −0.05 and 0.90 − 0.75 = 0.15. Thus D+ = 0.30, D− = 0.15 and D = 0.30. The largest gap occurs just after the empirical jump at 0.45. The value 0.30 is the statistic, not a p-value. A significance conclusion requires the correctly calibrated null distribution and a stated testing rule.

Illustrative case study

Fictional case study: Linden Data compares transaction-duration distributions before and after a system change. A KS test flags a difference, and the team plots both empirical CDFs to locate it. The main change affects moderate durations rather than the rare slowest cases that matter most to its service target. Analysts therefore add targeted tail summaries and investigate repeated transactions from the same accounts. The test provides evidence of a distributional difference but does not by itself identify the operational cause or answer every performance question.

Common misconceptions and mistakes

Common mistakes include fitting a normal mean and variance to a sample and then using the ordinary fully specified KS p-value unchanged. Parameter estimation changes the reference distribution of the statistic; an appropriate adjusted test or calibrated bootstrap is needed. Standard continuous-distribution formulas also do not automatically apply to discrete data or substantial ties. Dependence between observations can invalidate ordinary calibration. Failure to reject is not proof that distributions are identical. KS is often less sensitive to tail differences than methods designed specifically for tail assessment.

Business use

Businesses can use KS as one component of distribution monitoring and model checking. Report which form of the test was used, the reference distribution or comparison sample, sample sizes, statistic and calibration method. Inspect the practical size and location of the discrepancy rather than relying on a p-value threshold alone. When monitoring many features or repeatedly testing over time, account for the resulting multiplicity and dependence. Choose complementary summaries that match the decision, such as tail exceedance rates or quantile differences.

Check your understanding

Why does the worked calculation use both i/n and (i − 1)/n?

Answer: The empirical CDF jumps at each observation. Comparing the reference CDF with values on both sides of those jumps is necessary to find the maximum absolute gap.

Related terms

Goodness-of-fit
empirical CDF
two-sample test
Lilliefors test
Anderson–Darling test
bootstrap calibration
distribution shift

Learn next

plot the empirical and reference CDFs for the worked sample, then compare results when the reference parameters are known versus estimated. Explore how ties, dependence and tail-focused questions affect the choice of test and calibration.

Sources

Sources: SciPy, kstest documentation, https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kstest.html. NIST/SEMATECH e-Handbook, Kolmogorov–Smirnov Goodness-of-Fit Test, https://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm. Worked sample and fictional case are original illustrations.

Previous: Kolmogorov's axioms Next: Krichevsky - Trofimov estimator