Does every sample require equal selection probabilities?

No; unequal probabilities are common in valid designs.

Is a training example a sample?

That usage is common in machine learning, but it does not describe the dataset's selection design.

Can a non-random sample be useful?

Yes, provided conclusions respect its purpose and limitations.

AI Glossary / Statistics and probability / Intermediate

Sample

Definition

A sample is a collection of units or observations selected from a population or generated by a specified data-collection process. It provides information about the observed cases and may support inference about a wider population when the design and assumptions justify that inference. A sample need not be random, independent, or representative merely because it contains many records.

What it means

Organizations often cannot examine every customer, transaction, product, or possible operating condition. A sample supplies a manageable body of evidence, but what it can establish depends on how the observations entered the dataset.
In statistics, a random sample often has a precise probabilistic meaning. In machine learning, a sample can also mean one individual training example. These usages should be distinguished: a dataset of many training examples is not necessarily a probability sample of future users or deployment conditions.

How it works

Define the target population and the unit of analysis. A customer, account, order, and interaction are different units. If some customers generate many interactions, an interaction-level dataset does not automatically represent the customer population equally.
Identify the sampling frame or data-generating source. A customer register, a sensor stream, or a public website may omit relevant units. The population of interest and the accessible frame need not coincide.
Describe the selection method. Probability sampling uses a random mechanism with known inclusion probabilities under the design. Simple random sampling is one form; stratified, clustered, and multistage designs offer others. Probability sampling does not require every unit to have the same probability.
Non-probability samples can include volunteers, convenience cases, or deliberately selected examples. They can be useful for exploration and other purposes, but population inference requires additional assumptions or modeling rather than an unsupported claim of random representativeness.
Distinguish selected units from responding or usable units. Nonresponse, failed measurements, and exclusions change the available evidence. Record their reasons and examine whether they are related to the quantities being studied.
Use an analysis that reflects the design. Unequal inclusion probabilities may require weights, and clustering can change uncertainty. Treating thousands of correlated observations as independent can exaggerate precision.
For AI development, separate training, validation, and test samples in a way that matches the intended deployment. Near-duplicates, repeated users, and future information can create leakage across splits. Random row splitting is not sufficient for every dataset.
Report sample size alongside scope, dates, selection, exclusions, and limitations. A large sample can reduce random uncertainty under suitable conditions but cannot automatically repair missing population groups, systematic measurement error, or a mismatch between historical and future conditions.

Three practical examples

Customer research: a company selects accounts from a current register to estimate service satisfaction.
Quality assurance: inspectors examine selected units from a production lot under a documented sampling plan.
AI evaluation: researchers assemble a test dataset intended to reflect the inputs and users the deployed system will encounter.

Think of it this way

Think of tasting a portion of a mixture. The portion can tell you something useful only if you understand how it was taken and whether the mixture is uniform. Taking a larger scoop from the same unrepresentative corner does not necessarily reveal what is happening elsewhere.

Worked example

A fictional company has a complete register of 3,000 eligible accounts. It selects 300 by simple random sampling without replacement, and every selected account responds. Each account has inclusion probability 300/3,000 = 0.10 and a design weight of 10.
Among the sample, 120 accounts report using a feature. The sample proportion is 120/300 = 0.40, or 40%. The corresponding expansion estimate of the population total is 120×10 = 1,200 accounts.
The 1,200 is an estimate, not an observed count of all users. Another valid random sample could give a different result. If the 300 accounts had instead volunteered after seeing a promotional message, the same arithmetic would describe the respondents, but the simple-random-sampling justification for generalizing would be absent.

Illustrative case study

Fictional case: Pinebridge Software evaluates a new support classifier using tickets from its busiest customers. Initial results look strong, but the sample contains few small accounts and almost no tickets in a recently added language.
The team documents the limited scope and supplements evaluation with cases reflecting the intended deployment population. It keeps a record of selection and exclusions rather than assuming that the large number of existing tickets makes the assessment representative.

Common misconceptions and mistakes

A sample is not synonymous with a random sample.
More rows do not necessarily mean more independent information: repeated measurements and duplicates matter.
Matching a few demographic proportions does not prove absence of selection bias.
A census of an incomplete database is still incomplete for a wider target population.
Randomly dividing a biased dataset into training and test portions does not remove the original selection problem; it mainly preserves that dataset's composition in both portions.

Business use

Before using a sample to support a business decision, ask who or what could enter it, who is absent, and whether the selection process supports the intended conclusion. This turns sample documentation into a practical control on overstated forecasts, customer claims, and AI performance reports. Keep the population definition visible when results are reused for a new decision.

Check your understanding

Answer: A dataset contains 10,000 interactions from 20 customers.

Is its customer sample size 10,000?

Answer: No; there are 20 distinct customers, with repeated interactions.

Does the worked example observe 1,200 feature users?

Answer: No; it observes 120 and estimates the wider total under the stated design.

Related terms

Population
Sample size
Training set
Selection bias
sampling frame
probability sampling
non-probability sampling
inclusion probability

Learn next

Write a short sampling description for a dataset you use: target population, unit, frame, selection method, dates, response, and exclusions. Identify which conclusions follow directly from the observed cases and which require inference.

Population

Sources

Statistics Canada, Probability sampling: https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch13/prob/5214899-eng.htm Statistics Canada, Non-probability sampling: https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch13/nonprob/5214898-eng.htm The account example and case are original fictional illustrations.

Previous: Sammon mapping Next: Sample augmentation