Must every unit have the same chance of selection?

No. The probabilities can differ if they are known and the design and analysis account for them.

Is a voluntary online poll a probability sample?

Usually not merely because many people respond. Participation probabilities are generally not established by a defined probability-selection mechanism.

Does probability sampling guarantee representativeness in every realized sample?

No. Random variation remains, and coverage, nonresponse and measurement issues require separate attention.

AI Glossary / Data science and analytics / Intermediate

Probabilistic Sampling

Definition

Probabilistic sampling, usually called probability sampling, selects units through a random mechanism with known, positive inclusion probabilities for the population covered by the design. Those probabilities support statistical estimation and assessment of sampling uncertainty.

What it means

A sample is a subset used to learn about a larger population. Probability sampling makes the selection mechanism explicit instead of relying only on volunteers, convenient contacts or an analyst's judgment about which cases look typical.

The probabilities do not have to be equal. A design may deliberately sample a smaller subgroup more heavily to support useful subgroup estimates. What matters is that selection follows the stated random procedure and the relevant inclusion probabilities can be determined.

This gives a basis for inference, not a guarantee that one realized sample perfectly mirrors every population characteristic. Coverage gaps, nonresponse and inaccurate answers can still undermine conclusions. A carefully randomized selection from an incomplete list cannot automatically represent people who had no chance of appearing on that list.

How it works

Define the target population and the sampling frame used to reach it. Choose a design suited to the required estimates, available information and cost. Examples include simple random, stratified and multistage probability designs.

Use the specified random selection process and retain the information needed to calculate inclusion probabilities. In a basic unequal-probability analysis, a sampled unit's base weight is the inverse of its inclusion probability. Further adjustments may be needed for nonresponse or other features of the survey.

The estimator and uncertainty calculation should reflect the design. Treating a clustered or disproportionately stratified sample as if every row were an independent simple random draw can misstate precision or population quantities.

Document what happened after selection, including unsuccessful contacts, replacements and exclusions. Substituting an easy-to-reach person for a selected person without a valid rule can change the selection mechanism that justified the original inference.

Three practical examples

A customer research team randomly samples within service tiers, using different sampling fractions to obtain enough responses from a small tier.
An organization selects branches and then randomly samples records within selected branches to reduce collection cost.
An AI evaluation team draws a probability sample from a defined collection of incoming tasks to estimate performance for that task population.

Think of it this way

Think of a draw whose rules specify exactly who is eligible and each person's chance of selection. The rules allow you to understand how the selected group was produced.

The analogy has limits because real sampling can involve several stages and unequal probabilities. Knowing the draw rules also does not guarantee that selected people respond or that their answers are accurate.

Worked example

Suppose a fictional customer population contains 900 standard customers and 100 premium customers. The team independently selects simple random samples of 90 standard and 50 premium customers within those groups.

A standard customer's inclusion probability is 90/900 = 0.10, giving a base weight of 10. A premium customer's probability is 50/100 = 0.50, giving a base weight of 2.

Assume the observed sample means for a usage measure are 20 and 80 respectively, with complete responses. The unweighted sample mean is (90 × 20 + 50 × 80)/140, approximately 41.43. The population-share-weighted estimate is (900 × 20 + 100 × 80)/1,000 = 26.

The difference arises because premium customers were deliberately oversampled. The value 26 is an estimate under the design and assumptions, not a claim that the unknown population mean is exactly 26. An uncertainty calculation would also need information about within-group variability.

Illustrative case study

Illustrative fictional case: Pine Support wants to evaluate an AI assistant on all incoming requests. The team initially samples only requests that staff have already marked easy to review, then describes the result as a random evaluation of the entire workload.

A reviewer points out that difficult and incomplete requests had no chance of selection. The team defines the intended population, creates an appropriate frame and uses a documented probability design. It records which selected cases cannot be evaluated and investigates why.

The fictional lesson is that randomness inside a convenient subset does not remove the subset's coverage limitation. The case does not promise that probability sampling resolves every evaluation bias or establishes performance on future workloads.

Common misconceptions and mistakes

Do not confuse probability sampling with equal-probability sampling. Unequal probabilities can be intentional and useful when the analysis accounts for them.

Do not assume a random-number generator repairs an incomplete sampling frame. Avoid replacing nonrespondents informally or ignoring design weights when estimating population summaries. A large sample can reduce some sampling variability while leaving systematic coverage or response problems intact. Finally, random sampling and random assignment serve different purposes: selecting units and allocating treatments are separate operations.

Business use

Use probability sampling when the project needs defensible estimates about a defined population. Match the design to the important reporting groups and the resources available for collection and follow-up.

Maintain the frame, selection rules, inclusion probabilities and response outcomes. In AI evaluation, distinguish a population-based estimate from a deliberately difficult challenge set; both can be useful, but they answer different questions. Reports should identify the population actually covered and explain uncertainty without suggesting that the sampling design eliminates all other sources of error.

Check your understanding

A team randomly selects 100 records from the easiest 500 cases and claims to represent all 10,000 cases. What is the problem?

Answer: Cases outside the easy subset had no chance of selection. Random selection within that subset does not justify an unrestricted claim about the full population.

Related terms

Simple Random Sampling
Stratified sampling
Cluster Sampling
Sampling error
Sampling frame
inclusion probability
survey weight
nonresponse bias

Learn next

Read sampling frame and stratified sampling. Define the population for one evaluation, identify who could be missed by its frame and calculate inclusion probabilities for a small design. Explain how the analysis should change if one subgroup is oversampled.

Stratified sampling

Sources

University of Chicago, probabilistic-sampling glossary entry: https://ds1.datascience.uchicago.edu/glossary.html

Statistics Canada, probability-sampling designs: https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch13/prob/5214899-eng.htm

Statistics Canada, sample design and inclusion probabilities: https://www150.statcan.gc.ca/n1/pub/12-539-x/2009001/sample-plan-eng.htm

Statistics Canada, sample planning and nonsampling considerations: https://www150.statcan.gc.ca/n1/pub/12-539-x/steps-etapes/4058330-eng.htm

The customer calculation and support evaluation are fictional illustrations.

Previous: Probabilistic roadmap Next: Probabilistic semantics