Is exclusion bias the same as a small sample?

No. A small sample may be uncertain without systematically excluding a relevant group; a large sample can still have exclusion bias.

Can data cleaning create it?

Yes, if removal rules disproportionately discard relevant cases and the effect is ignored in analysis or reporting.

Must every study include everyone?

No. Studies need a defined scope. The key is whether the evidence supports the intended population-level claim and whether exclusions are justified and transparent.

AI Glossary / Governance, ethics and safety / Intermediate

Exclusion Bias

Definition

Exclusion bias arises when relevant people, groups, or cases are left out of data collection, testing, or analysis in a way that distorts the resulting evidence. In AI evaluation, it can make a system appear more broadly suitable than the evidence supports. The concern is not every exclusion, but exclusions that undermine the intended claim or systematically overlook affected users.

What it means

A test can be rigorous for the participants it includes while saying little about those who could not participate. An AI interface study that recruits only people with a particular device, language, or work schedule may omit users expected at launch. Exclusions can also happen during cleaning or analysis: incomplete records, difficult cases, or failed sessions may be removed without examining why they are missing. The result can present the experience of successful participants as if it represented everyone who tried to use the service.

How it works

Review the path from the intended population to the final analyzed sample. Identify who could enter, who was invited, who participated, and whose results were retained. For each exclusion, document its reason and relationship to the study purpose. A justified scope restriction should narrow the conclusion accordingly. If omitted cases are relevant, seek suitable evidence, revise recruitment or test access, and examine missingness. Exclusion bias overlaps with selection and representation bias; the practical task is to locate the omission and explain how it limits the inference or hides a consequence.

Three practical examples

An assistant is evaluated only by staff who use a mouse, while keyboard-only users are expected to rely on the finished service.
A model report drops all records with failed document extraction, then presents accuracy as if it covered every document submitted.
A pilot recruits only daytime workers even though the service will be used across shifts with different tasks and connectivity. These omissions can affect both statistical evidence and understanding of usability.

Think of it this way

Think of reviewing a concert venue by interviewing only people who managed to get through its entrance. Their experience matters, but it cannot reveal all barriers faced by people who could not enter. The analogy is limited because dataset exclusions may be invisible and occur through several technical steps. It makes clear why successful participation should not be the sole condition for inclusion in an evaluation of access.

Worked example

Suppose one hundred users attempt a task with an AI service. Twenty cannot complete the initial input step and are excluded from the final performance report. Of the eighty remaining users, seventy-two complete the full task successfully. Reporting 72/80 gives a 90 percent success rate among users who passed the initial step. Reporting successful completion among all one hundred attempts gives 72 percent. Both calculations are arithmetically correct, but they answer different questions. If the report claims end-to-end success for all users, excluding the initial failures makes the claim misleading. The team should report the stage-specific result and the full-workflow result, then investigate who experienced the twenty initial failures. These numbers alone do not show whether a demographic group was disproportionately affected, but they identify an evidence gap that matters.

Illustrative case study

Fictional case: Birch Documents tests an AI form assistant using completed sessions. Reviewers notice that several users abandoned the session before any model output was generated, so those attempts never entered the benchmark. The team expands the evaluation to include the input process and investigates whether instructions, formats, or assistive-technology support contributed. It reports completion and model accuracy separately. This invented case illustrates a more complete evaluation boundary without claiming that every abandonment was caused by the AI or that one fix resolved the problem.

Common misconceptions and mistakes

Not every exclusion is inappropriate. A clearly defined study may legitimately focus on a specific population, provided the conclusion does not silently expand beyond it. More data from the already included group does not necessarily repair missing coverage. Adding one representative person does not establish performance for an entire diverse population. A low number of complaints also does not show that excluded users had no problems; they may have lacked access to the feedback channel. Finally, missing cases should not automatically be assigned successful or unsuccessful labels without evidence about what happened.

Business use

Define the population and workflow that the evaluation is meant to cover. Keep a record of inclusion rules, exclusions, failed starts, missing outcomes, and analysis decisions. Make recruitment and testing accessible where relevant, and consult users whose needs may differ from the development team's assumptions. Report denominators at each stage so omitted cases remain visible. When coverage is incomplete, qualify the deployment claim and plan appropriate follow-up rather than hiding the gap in an average. Review automated cleaning rules because they can systematically remove exactly the difficult cases that need investigation.

Check your understanding

A benchmark removes every failed upload before computing service success. What must a reader know before interpreting the score?

Answer: The score applies only after upload succeeded. Failed uploads and the full-attempt denominator are needed to assess end-to-end service success.

Related terms

Learn next

Draw a flow diagram from all intended users or submitted cases to the final evaluation sample. Count what is removed at each step and record why. Identify one exclusion that could change the conclusion and design a way to investigate it.

Sources

NIST SP 1270, exclusion-bias glossary: https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934464 W3C Web Accessibility Initiative, Involving Users in Evaluating Web Accessibility: https://www.w3.org/WAI/test-evaluate/involving-users/ Suresh and Guttag, sources of harm in the ML life cycle: https://arxiv.org/html/1901.10002v5

Previous: Example-based machine translation Next: Excursion probability