Can a valid probability design oversample a small group?

Yes, with appropriate probabilities and analysis.

Is stratification the same as clustering?

No; stratification samples within groups, while cluster designs select groups as part of the sampling process.

Does a random train-test split define a population sampling design?

No; it partitions the available data without explaining how those data entered the study.

AI Glossary / Statistics and probability / Intermediate

Sampling design

Definition

Sampling design is the set of rules specifying how units are selected from a population or sampling frame. It includes choices such as sampling units, stages, strata, allocation, and selection probabilities. In probability sampling, the design defines a probability distribution over possible samples. Broader usage may also include estimation planning, so the scope should be stated.

What it means

A sampling design explains why the observed cases entered a study and what other samples could have been selected. That explanation supports appropriate estimates and uncertainty calculations.
The same number of observations can carry different information under different designs. Selecting scattered individual units, selecting whole clusters, or sampling separately within meaningful strata can produce different costs, coverage, and precision. A large sample size cannot substitute for a defensible selection process.

How it works

Define the target population and the quantities to be estimated. A study of all accounts may require a different design from one intended to compare regions or estimate outcomes for a rare subgroup.
Identify the frame and sampling unit. Check whether the frame adequately covers the population and whether records map to the intended units. Establishments, enterprises, households, and individuals may require different selection stages.
Choose a selection structure. Simple random sampling gives equal probability to samples of a fixed size. Stratified sampling selects within defined subgroups. Cluster and multistage designs select groups and possibly units within groups, often reducing collection costs.
Set allocation and probabilities. Proportional allocation can preserve population shares, while disproportionate allocation may improve information for small domains. Unequal inclusion probabilities are compatible with probability sampling when they are known and handled appropriately.
Plan estimation and uncertainty alongside selection. Design weights may be the inverse of inclusion probabilities, and variance calculations may need stratum and cluster information. An analysis that ignores the design can give misleading estimates or standard errors.
Specify replacement, coordination, and repeat-survey rules where relevant. Sampling with replacement differs from sampling without replacement. Rotating panels deliberately create overlap across periods, which changes dependence and the analysis of change.
Plan field implementation, response follow-up, and records of departures. Replacing a nonrespondent with a convenient volunteer can change the effective selection process. Nonresponse and frame errors remain concerns even when the original random selection is well designed.
Document the design clearly enough to reproduce selection and interpret results. For AI evaluation, purposeful challenge sets and probability samples can both be useful, but they support different claims. A stress-test collection designed to expose failures is not automatically an estimate of failure frequency in deployment.

Three practical examples

Market research: a survey samples separately within regions to support regional estimates.
Field inspection: a multistage design selects sites and then products within selected sites to control travel costs.
AI evaluation: a team distinguishes a representative usage sample from a deliberately difficult diagnostic benchmark.

Think of it this way

Think of planning a route through a landscape before taking measurements. Different routes can cover the same number of stops but reveal different parts of the landscape. The route rules determine what can be inferred from the stops and how confidently the resulting map can be interpreted.

Worked example

A fictional population contains 600 units in stratum A and 400 in stratum B. Suppose a stratified sample selects 60 from A and 40 from B, independently by simple random sampling within strata. Both inclusion probabilities are 0.10, so all design weights are 10.
If observed stratum means are 20 and 30 units, the stratified estimate of the population mean is 0.6×20+0.4×30 = 24. Here the raw combined sample mean also equals 24 because allocation is proportional.
Now consider a different design selecting 50 from each stratum and suppose the same stratum means are observed. Design weights become 600/50 = 12 and 400/50 = 8. The raw mean is 25, but the properly population-weighted estimate remains 24.
This illustrates allocation and estimation, not a claim that both designs have equal precision. Variability within strata and the full design determine that comparison.

Illustrative case study

Fictional case: Fernbridge Services wants reliable AI-assistant satisfaction estimates for several small regions. A simple proportional sample would leave very few observations in those regions, so analysts plan a stratified sample with additional regional coverage.
They retain inclusion probabilities, use appropriate weights for the overall estimate, and calculate uncertainty using the design. The regional oversampling is described as an intentional design choice rather than hidden as though the raw respondent mixture matched the customer population.

Common misconceptions and mistakes

Sampling design is not just the sample size.
Equal inclusion probabilities do not by themselves describe every aspect of a joint sampling design.
Random selection and random treatment assignment solve different problems: selecting units supports population inference, while assigning interventions supports causal comparison under its conditions.
Clustered observations are not automatically independent.
A probability design does not eliminate measurement error, nonresponse, or frame undercoverage.

Business use

Use sampling design to align data collection with the decisions the evidence must support. Plan subgroup requirements, cost, selection, weighting, and uncertainty together. In AI projects, identify whether a dataset is meant to estimate ordinary deployment performance, compare interventions, or discover weaknesses; one selection strategy rarely serves every objective equally well.

Check your understanding

Why does the example's equal-allocation raw mean differ from the population-weighted mean?

Answer: The sample gives the two strata equal shares while the population shares are 60% and 40%.

Can we decide which allocation is more precise from the means alone?

Answer: No; within-stratum variation and the design are needed.

Related terms

Stratified sampling
Cluster Sampling
Sample rotation
Sampling frame
inclusion probability
design weight
sampling variance

Learn next

Write a sampling plan for one business question, including the population, unit, frame, selection stages, allocation, estimator, and variance method. Identify likely implementation departures before data collection begins.

Data collection

Sources

Statistics Canada, Methodology of the Canadian Labour Force Survey, Chapter 2 Sample design: https://www150.statcan.gc.ca/n1/pub/71-526-x/2017001/chapt2-eng.htm OECD Glossary of Statistical Terms, Sample design: https://www.oecd.org/content/dam/oecd/en/publications/reports/2008/09/oecd-glossary-of-statistical-terms_g1gh9ad7/9789264055087-en.pdf The stratified numerical example and case are original fictional illustrations.

Previous: Sampling bias Next: Sampling distribution