Is selection bias the same as sampling bias?

The terms overlap, but selection bias is often used more broadly for mechanisms governing inclusion or observation.

Does a random split of existing records prevent it?

No. The split can preserve the same selected population in every subset.

Can selected data ever be useful?

Yes. It may answer a bounded question, and some broader quantities can be recovered under justified assumptions and sufficient information.

AI Glossary / Governance, ethics and safety / Intermediate

Selection bias

Definition

Selection bias is distortion in an inference caused by the process determining which observations are included in the analysis. The observed cases can differ systematically from the cases needed to answer the question. In AI, selection can affect training, evaluation, feedback, and apparent relationships between features and outcomes.

What it means

The available data may describe people who used a service, completed a process, received a decision, or generated a visible outcome. Those are selected groups. A conclusion about everyone eligible for the service may require information that the selected records do not contain.
Selection can do more than change population proportions. In some situations, conditioning on a shared consequence of two factors creates an association between them even when they were unrelated in the original population. This is one reason selection bias matters for causal interpretation as well as predictive generalization.

How it works

Define the inference you want: a population average, model performance, an association, or a causal effect. Identify the rule governing inclusion and the information available about excluded units. Consider whether selection depends on causes or consequences of the variables being analyzed.
Use subject-matter knowledge and, where helpful, a causal diagram to express these assumptions. Redesigned collection, external population information, or appropriate statistical methods may help. However, recoverability depends on what is observed and what assumptions are justified. A method cannot generally reconstruct arbitrary missing information from a selected sample without additional support.

Three practical examples

A retention model learns only from customers who remain long enough to complete a follow-up survey, omitting early departures.
A product analyst studies only listings that received a click, then interprets relationships within clicked listings as if they described all listings.
An AI evaluation reports accuracy only on requests the system chose to answer, without reporting deferred requests or the selection rule.

Think of it this way

Think of judging a city's weather by photographs chosen for a tourism brochure. The selected images may be genuine, yet the selection process changes what conclusions they support. The analogy is limited because scientific selection mechanisms can be subtle and mathematically structured; identifying attractive images is much easier than identifying all pathways that affect dataset inclusion.

Worked example

Consider a fictional population of 100 items. Each has two binary features, A and B. There are 25 items in each of four combinations: neither feature, A only, B only, and both. In the full population, B is present in 50% of items with A and 50% of items without A.
Now retain only items with A or B. The 25 items with neither are excluded. Among the 50 retained items with A, 25 have B, so the rate remains 50%. Among the 25 retained items without A, all 25 have B, so the rate is 100%. A negative association appears within the selected dataset even though the original population had none. Selection depended on the shared result of A and B; the numbers illustrate a collider-selection mechanism, not a causal effect of A on B.

Illustrative case study

At fictional Elm Marketplace, analysts examine only products that received user attention. They notice that one attractive feature appears negatively associated with another and consider removing one from a recommendation model. A review shows that products entered the dataset when either feature helped attract attention.
The team compares the selected records with the broader listing population and revisits the question before interpreting the pattern. This fictional case illustrates how selection can change associations. It does not imply that every negative correlation is a selection artifact or that a diagram alone proves the correct explanation.

Common misconceptions and mistakes

Selection bias is not just a small-sample problem. It can persist in a very large dataset. It is also not identical to confounding, although both can distort causal conclusions and can occur together.
Do not assume that controlling for more variables always helps. Conditioning on a variable involved in selection can sometimes introduce rather than remove a spurious association. Likewise, comparing observed groups after matching does not automatically resolve unobserved selection processes. A descriptive result can be valid for the selected group while being invalid when generalized to a different target; the claim must identify its population.

Business use

This concept helps businesses interpret service metrics and evaluate AI pilots honestly. Reports should describe the denominator, inclusion rules, missing outcomes, and intended population. For systems that abstain or route difficult cases to people, report both the performance on answered cases and the proportion and characteristics of cases answered. Otherwise, an apparent quality improvement may simply reflect selecting easier work.

Check your understanding

A system's answered-case accuracy rises after it begins refusing difficult requests. Can the team conclude that its performance improved for all requests?

Answer: Not from that number alone. The set being evaluated changed. Coverage, deferred-case outcomes, and the intended service objective must also be examined.

Related terms

Learn next

Write the inclusion rule for a dataset as a sentence beginning with a record appears only if. Then list factors that influence that condition. This often reveals whether the analysis is about the target population or a selected subgroup and where additional data or assumptions are needed.

Sources

Sources: Google Machine Learning Glossary, selection bias: https://developers.google.com/machine-learning/glossary#selection-bias . Bareinboim, Tian, and Pearl, Recovering from Selection Bias in Causal and Statistical Inference: https://ojs.aaai.org/index.php/AAAI/article/view/9074 . Author-hosted paper: https://ftp.cs.ucla.edu/pub/stat_ser/r425.pdf . NIST SP 1270: https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934464 .

Previous: Selection And Sampling Bias Next: Selection problem