AI Glossary / Statistics and probability / Intermediate
Sampling distribution
Definition
A sampling distribution is the probability distribution of a statistic across the samples that could arise from a specified population and sampling process. It describes how an estimator or other sample summary varies before a particular sample is observed. It is different from the distribution of individual data values within one sample.
What it means
Imagine repeating the same study design many times and calculating the same statistic each time. The pattern of those possible statistic values is its sampling distribution. The repeated studies are a conceptual device; an actual analysis often uses a theoretical or estimated distribution based on one collected sample.
A mean, proportion, median, variance, and regression coefficient each have their own sampling distribution. The distribution depends on the statistic, population, sample size, and selection mechanism.
How it works
Specify the population or probability model and the sampling design. Sampling with replacement, sampling without replacement, clustered sampling, and dependent time-series observation can give different distributions even for the same nominal sample size.
Define the statistic exactly, including weights and missing-data handling. Changing the estimator changes the sampling distribution. A raw average and a survey-weighted average generally do not share the same uncertainty calculation.
Identify its center, spread, and shape. The expected statistic value can reveal bias relative to a target parameter. Its standard deviation is the standard error. Its shape affects how intervals and hypothesis tests should be constructed.
For small finite examples, enumerate all possible samples and their probabilities. Equal weighting of possible samples is justified only when the design makes them equally likely. The same statistic value may arise from several distinct samples.
For some models, derive the distribution analytically. For example, an average of independent normal observations is exactly normal. Other statistics may have different exact distributions, and large-sample approximations need their own conditions.
Simulation can approximate a sampling distribution by repeatedly generating data under a specified model and applying the intended analysis. It is useful for complex estimators, but the result reflects the simulated assumptions rather than proving those assumptions true.
Bootstrap resampling uses the observed sample as a basis for approximating uncertainty. The resampling method must reflect relevant dependence and design features. A bootstrap is not the exact population sampling distribution and can perform poorly in unsuitable settings.
Use the distribution to assess uncertainty in estimates or calibrate tests. A null distribution is a sampling distribution under a specified null model; it need not be the distribution under every alternative. Distinguish uncertainty from sampling variation from systematic errors in measurement, selection, or model definition.
Three practical examples
- Survey reporting: analysts use the sampling distribution of an estimated proportion to quantify uncertainty.
- A/B testing: a reference distribution helps assess how much a group difference could fluctuate under a defined null model.
- AI benchmarking: researchers examine variation in a performance statistic across appropriately sampled evaluation datasets.
Think of it this way
Think of repeating a measurement procedure that produces one summary number each time. A histogram of those summary numbers represents the sampling distribution. A histogram of all individual readings answers a different question, because the summary calculation changes both the scale and the variation.
Worked example
Take an original toy population with values 0, 2, and 4, each equally likely on every draw. Draw two observations independently with replacement and calculate their mean. There are nine equally likely ordered pairs.
The possible sample means are 0, 1, 2, 3, and 4. Their probabilities are respectively 1/9, 2/9, 3/9, 2/9, and 1/9. For instance, a mean of 2 arises from (0,4), (2,2), and (4,0).
The expected sample mean is 2. Its variance is [1×4+2×1+3×0+2×1+1×4]/9 = 12/9 = 4/3. Its standard error is sqrt(4/3), approximately 1.155 units.
The population variance is 8/3, so the mean's variance is half as large, consistent with independent averaging of two observations. This exact sampling distribution is discrete and is not normal.
Illustrative case study
Fictional case: Cedar Evaluation compares an AI model's accuracy on successive small samples of support requests. The reported percentages fluctuate, and managers initially interpret every movement as a system change.
The analyst uses a sampling model suited to the evaluation process to show the expected variation in the score. The team then distinguishes plausible sampling fluctuation from changes requiring investigation, while separately checking whether the request mix or labels have shifted.
Common misconceptions and mistakes
A sampling distribution is not the same as the histogram of one dataset.
It can be studied even when only one actual sample is collected, using assumptions, theory, or appropriate resampling.
Not every sampling distribution is normal, and a fixed sample-size cutoff does not make all approximations valid.
Standard error is not the standard deviation of the raw observations.
A narrow sampling distribution does not rule out systematic bias or poor population coverage.
Business use
Use sampling distributions to explain why measured results vary and how much confidence a decision can place in an estimate. In business reporting, this helps prevent overreaction to small changes while exposing when data are too limited for a precise comparison. Preserve the design assumptions so the uncertainty statement remains interpretable.
People also ask
Does every statistic have the same sampling distribution?
No.
Is a bootstrap distribution always exact?
No; it is an approximation based on observed data and resampling assumptions.
Does collecting a larger sample always make the statistic normally distributed?
No; the statistic and conditions matter.
Can a sampling distribution be biased?
Its center can differ from the target parameter, indicating estimator bias.
Check your understanding
Why does the toy mean of 2 have probability 3/9?
Answer: Three ordered pairs produce it.
Why is the sampling distribution narrower than the population distribution here?
Answer: Independent averaging reduces variance by the sample size.
Is this a distribution of individual draws or of two-draw means?
Answer: It is a distribution of the means.
Related terms
Learn next
Enumerate possible samples from a tiny population and calculate two different statistics. Compare their sampling distributions, then simulate the same design to see how the numerical approximation approaches the exact result.
More in Statistics and probability
- Sample variance - Sample variance is a measure of how widely observations are dispersed around their sample mean.
- Sample weighting - Sample weighting assigns different numerical contributions to observations when calculating estimates, fitting models, or evaluating performance.
- Sampling design - Sampling design is the set of rules specifying how units are selected from a population or sampling frame.
- Sampling distribution of differences between means - The sampling distribution of differences between means is the probability distribution of a difference between sample averages over repeated samples drawn...
- Sampling distribution of the mean - The sampling distribution of the mean is the probability distribution of the averages produced by repeated samples under a specified population model and...
- Sampling error - Sampling error is the random variation in an estimate that arises because information is collected from a sample rather than the entire target population.
Sources
Howell, Statistical Methods for Psychology glossary, Sampling distributions: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html OpenStax, The Central Limit Theorem for Sample Means: https://openstax.org/books/introductory-statistics-2e/pages/7-1-the-central-limit-theorem-for-sample-means-averages The nine-pair example and case are original fictional illustrations.
Previous: Sampling design Next: Sampling distribution of differences between means