Does incompleteness always cause bias?

No, but it can, and it usually reduces available information.

Can imputation restore the original observations?

It supplies replacements under assumptions; it does not recover unknown facts merely by filling cells.

Should the historical term be used in a report?

Prefer the specific mechanism, such as lost records or attrition, while explaining this glossary meaning when relevant.

AI Glossary / Statistics and probability / Intermediate

Defective sample

Definition

A defective sample, in the historical statistical terminology used by the supplied glossary, is a sample made incomplete because the intended inquiry was not fully carried out. Selected units may not have been examined, records may have been lost, or observations may have become unavailable. The term describes incomplete realization of the planned study. It does not mean simply that a sample contains defective manufactured products.

What it means

The planned sample and the analyzed sample can differ. A study may select an appropriate set of units but fail to obtain usable information for every selected unit. The retained observations may then provide a different picture from the intended sample.

Contemporary reports usually describe the specific issue more precisely, such as nonresponse, attrition, missing records, or incomplete follow-up. That language helps identify what happened and which assumptions an analysis requires.

Incompleteness does not automatically make every conclusion unusable, nor does it imply a known amount or direction of bias. Its consequences depend on which observations are missing, why they are missing, and how the analysis handles that loss. Both reduced precision and systematic distortion can matter.

How it works

Compare the sampling or assignment records with the observations actually obtained. Identify which units or fields are absent and distinguish data loss from legitimate exclusions, ineligibility, and events that were meant to be recorded as outcomes.

Investigate whether missingness relates to known characteristics or likely outcomes. Compare available information about retained and missing units where possible. Recovery, follow-up, weighting adjustments, imputation, or explicit missing-data models may be appropriate, but each relies on design information and assumptions.

Report the planned sample size, observed sample size, reasons for loss, and analysis choices. Sensitivity analysis can show how conclusions change under plausible missing outcomes. If an event makes an outcome undefined rather than merely unobserved, the target question may need clarification; mechanically filling a blank is not necessarily meaningful.

Three practical examples

A selected customer survey loses some response files during processing, leaving an incomplete set of intended observations.
A longitudinal business study cannot obtain follow-up measurements from some participating firms, requiring investigation of attrition.
An AI benchmark omits difficult cases because a processing stage failed, making the analyzed set different from the originally assigned evaluation set.

Think of it this way

Think of a planned inventory in which several shelves were never counted and some tally sheets disappeared. The remaining sheets may be accurate, but they do not automatically describe the complete planned inventory. Counting more items on the already recorded shelves does not necessarily repair the missing coverage.

Worked example

Consider an original evaluation plan containing 100 assigned tasks. A processing failure leaves usable results for only 70 tasks. Among those, the AI system succeeds on 56 and fails on 14. The observed success rate is 56/70 = 80%.

Suppose the 30 missing outcomes are genuinely unknown and success is binary for every assigned task. The lowest possible success rate across all 100 tasks occurs if every missing task failed: 56/100 = 56%. The highest occurs if all 30 succeeded: (56+30)/100 = 86%.

Thus the available data alone bound the full assigned-set rate between 56% and 86%. These are logical bounds under the stated setup, not a confidence interval or an estimate of performance on a wider population. Assuming that missing tasks behave like observed tasks would produce an 80% estimate, but that requires a missingness assumption.

If records reveal that failures occurred mainly on long, complex tasks, the observed 80% may be especially misleading. If instead the evaluation protocol defines a system timeout as failure, those timeouts should be recorded as failures rather than relabeled missing. The target metric and failure definition must be settled before interpreting the denominator.

Illustrative case study

In this fictional case, Harbor Benchmarking compares two document-processing systems. A file-conversion error excludes several large documents from one system's result table. The remaining results appear easier to summarize, so an initial report overlooks the missing cases.

The team reconciles task identifiers against the assigned list, investigates the conversion problem, and distinguishes infrastructure data loss from system failures defined by the evaluation protocol. It reruns recoverable cases and reports unresolved omissions explicitly.

The invented case illustrates sample accounting. The remedy begins with understanding why observations disappeared, not with assuming that the available subset is interchangeable with the planned one.

Common misconceptions and mistakes

A defective sample is not synonymous with an ordinary small sample or with a sample of faulty goods. The relevant issue here is incomplete execution or loss of intended observations.

Increasing the number of retained observations does not automatically eliminate systematic bias caused by missing units. Likewise, a low nonresponse rate is reassuring only in context; a small missing subgroup may be particularly important for the outcome.

Do not automatically classify every death, closure, or withdrawal as an ordinary missing value. Such events can themselves be outcomes, competing events, or reasons that a later measurement is undefined. The appropriate treatment depends on the study question. Explicit event definitions prevent the analysis from confusing absence of data with absence of an event.

Business use

Businesses should maintain a complete accounting of selected records, collection attempts, processing failures, exclusions, and final analysis membership. Unique identifiers and reconciliation checks make unexplained losses easier to detect.

For AI evaluation, ensure that preprocessing does not silently remove the hardest inputs. Compare exclusions across systems and report the denominator used for every metric. If conclusions depend on assumptions about missing observations, communicate that dependence and examine alternatives. A polished dashboard cannot compensate for an undocumented gap between the planned study and the data actually analyzed.

Check your understanding

Check your understanding: Why is 80% only the observed rate?

Answer: Thirty assigned tasks have unknown outcomes.

What do 56% and 86% represent?

Answer: Extreme full-set rates under the binary-outcome assumptions.

Are they a confidence interval?

Answer: No.

Why distinguish a timeout from data loss?

Answer: The evaluation protocol may define the timeout as an observed failure.

Related terms

Learn next

Distinguish missing completely at random, missing at random, and missing not at random. Examine how weighting and sensitivity analysis depend on the available auxiliary information.

Sensitivity analysis

Sources

Sources: OECD Glossary of Statistical Terms, Defective sample, https://www.oecd.org/content/dam/oecd/en/publications/reports/2008/09/oecd-glossary-of-statistical-terms_g1gh9ad7/9789264055087-en.pdf. Australian Bureau of Statistics, Types of error, https://www.abs.gov.au/statistics/understanding-statistics/statistical-terms-and-concepts/types-error. The historical scope is explicitly retained; the benchmark bounds and fictional case are original.

Previous: Defective matrix Next: Defense in depth (computing)