Is an online survey necessarily non-probability based?

No. Online questionnaires can be administered to participants originally selected through a probability design.

Can these samples estimate population quantities?

Sometimes, using explicit models and external information, but validity depends on assumptions beyond the observed sample alone.

Can I quote a conventional margin of error?

A probability-sampling formula is not justified solely by the number of volunteer responses. Any uncertainty statement needs a defensible methodological basis.

AI Glossary / Statistics and probability / Beginner

Non-random (non-probability) sample

Definition

A non-random, or non-probability, sample is a collection of units selected without a probability sampling design that gives population members known, positive inclusion probabilities. Selection may depend on availability, voluntary participation, judgment, referrals, or quotas. Such a sample can provide useful observations, but the recruitment process alone does not justify treating its results as representative of an entire target population.

What it means

The distinction concerns how units enter the sample. Asking whoever visits a product forum produces a different evidential basis from selecting customers through a specified random mechanism. Forum users may have distinctive interests, experiences, or access to technology.

Non-probability does not mean that the data are fabricated, that every estimate is wrong, or that analysis is impossible. It means that population conclusions require additional assumptions and supporting evidence. A large number of responses can describe the participating group precisely while leaving substantial uncertainty about people who never participated.

How it works

Define the population and the question before choosing recruitment channels. Record where participants came from, eligibility rules, exclusions, and whether recruitment depended on recommendations or voluntary response. Distinguish the intended population from the population actually reachable through those channels.

Analyze the collected observations for their stated purpose. Exploratory interviews can reveal usability problems without estimating how common each problem is across all customers. Population estimates require stronger justification, potentially involving benchmark data, adjustment models, and sensitivity analyses.

Compare relevant characteristics with reliable population information where available. Weighting can align observed distributions, but it cannot automatically eliminate differences on unobserved characteristics. Report the assumptions behind any adjusted estimate. A random subsample drawn from an unrepresentative volunteer pool does not retroactively make the original recruitment probability-based for the wider population.

Three practical examples

A product team invites volunteers from its online community to explore difficulties using an AI assistant.
A researcher deliberately recruits experienced data managers to investigate how organizations document training datasets.
A business interviews available shoppers until predefined age-group quotas are filled, while acknowledging that selection within each quota remains non-random.

Think of it this way

Think of learning about a town by speaking to everyone who attends one particular club. You may hear detailed, valuable accounts, but membership of that club shapes whose experiences are present. Interviewing more club members does not necessarily reveal the experiences of residents who would never join.

Worked example

Consider an original teaching population of 1,000 customers. There are 200 frequent users and 800 occasional users. Suppose the true satisfaction rates are 90% and 50%, respectively. Total satisfied customers are 180 + 400 = 580, giving population satisfaction of 58%.

A volunteer survey contains 100 frequent and 100 occasional users. Assume, only for this illustration, that its satisfaction rates within each group exactly match the population rates. It records 90 + 50 = 140 satisfied respondents, or 70%. The unadjusted estimate exceeds the population value by 12 percentage points.

Weighting the two group rates by their known population shares gives 0.2 × 0.90 + 0.8 × 0.50 = 0.58. This correction works here because the within-group rates were assumed accurate. If occasional volunteers instead have satisfaction of 70%, the same adjustment gives 0.2 × 0.90 + 0.8 × 0.70 = 0.74. Matching group shares then leaves substantial bias. These constructed calculations demonstrate an assumption; they do not establish that weighting will succeed in an actual survey.

Illustrative case study

Fictional case study: Cedar Learning tests a new AI tutoring interface through an invitation posted in its enthusiast community. Volunteers give detailed feedback and identify confusing controls. The team uses those reports to improve the interface, which is consistent with its exploratory objective.

Management then proposes advertising the volunteer satisfaction percentage as the opinion of all customers. The analyst explains that enthusiasts and dissatisfied users may participate at different rates, and that customers outside the community were not recruited through a population sampling design. Cedar separates the usability findings from the proposed population claim and commissions a better-supported customer study. The fictional lesson is that a useful sample can support one decision without supporting every possible conclusion.

Common misconceptions and mistakes

“Thousands of responses make it representative.” Size does not remove systematic recruitment differences.

“Quotas make the selection random.” Quotas control selected counts; the recruitment method within the groups still matters.

“Weighting repairs every bias.” Adjustment depends on relevant information and assumptions about differences that remain after adjustment.

“No statistical analysis is allowed.” Descriptive and model-based analyses can be useful, provided their scope and assumptions are explicit.

Business use

Businesses frequently obtain non-probability data from feedback widgets, customer panels, support conversations, and voluntary surveys. These sources can help discover issues, generate hypotheses, and understand particular experiences. Their value increases when the intended decision is stated clearly.

Before presenting a population percentage, document recruitment, coverage, exclusions, response patterns, and the basis for any adjustment. For AI systems, also consider whether training or evaluation data overrepresent enthusiastic, connected, or highly active users. A benchmark score derived from a selected group should not silently become a promise about every deployment setting.

Check your understanding

A random 500 people are selected from a self-enrolled product panel. Does that automatically provide a probability sample of all customers?

Answer: No. Random selection within the panel does not resolve how customers entered the panel. Population inference must address that earlier selection process.

Related terms

Selection bias
Coverage error
Convenience sampling
Probability sample
quota sampling
survey weighting
target population

Learn next

Compare probability and non-probability recruitment, then study coverage and nonresponse separately. Next, examine how poststratification uses population benchmarks and which assumptions it leaves unresolved. Practice writing a conclusion whose population scope matches the recruitment evidence.

Sources

Australian Bureau of Statistics, Statistical terms and concepts glossary, non-random sample: https://www.abs.gov.au/statistics/understanding-statistics/statistical-terms-and-concepts/statistical-terms-and-concepts-glossary

Statistics Canada, Non-probability sampling, recruitment methods and inference limitations: https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch13/nonprob/5214898-eng.htm

The calculations and Cedar Learning case are original teaching illustrations, not reported survey findings.

Previous: Non-photorealistic rendering (NPR) Next: Non-randomness