Does sample size determine inclusion probability?

Only with additional design information. Under simple random sampling without replacement, it is n/N; the worked design deliberately differs.

Can a record have inclusion probability one?

Yes, if the design always includes it.

Is probability the same as the realized sampling fraction?

No. The fraction is an aggregate ratio; individual probabilities depend on the design.

AI Glossary / Statistics and probability / Advanced

Sampling probability

Definition

Sampling probability, in finite-population sampling, usually means a unit's inclusion probability: the probability that it appears in the sample under the specified sampling design. It is a property of the selection procedure. It need not be equal for all population units, and it is distinct from the probability of selecting a particular complete sample.

What it means

Consider an audit. A design may deliberately give records from some groups a higher chance of selection. To interpret the audit properly, analysts need to know those chances, not merely which records were eventually selected.

A selected record's inclusion indicator equals one, but its design probability does not retrospectively become one. The probability describes what would happen across repetitions of the selection procedure, with the population and design held fixed.

How it works

Let I_i indicate whether unit i is included. Its inclusion probability is pi_i = P(I_i = 1), which also equals the expectation of I_i. Calculate it by summing the probabilities of all possible samples containing that unit.

For a population total, the Horvitz - Thompson estimator adds y_i/pi_i over distinct sampled units. With known positive inclusion probabilities, its design expectation equals the population total. Joint inclusion probabilities describe whether two units appear together and matter for variance calculations. Selection probabilities alone do not resolve nonresponse or measurement errors.

Three practical examples

1. Record audits: analysts sample more heavily from a small, operationally important category. They retain the original inclusion probabilities so that a population estimate does not treat the oversampled category as naturally more common.

2. AI evaluation: a team samples examples from language groups at different rates. It distinguishes performance on the deliberately balanced evaluation set from an estimate for the original population of examples.

3. Customer research: invitations are randomly selected, but some invitees do not respond. The invitation probability describes selection, not the later probability of obtaining a usable response. The team records these stages separately.

Think of it this way

Think of a lottery that chooses bundles of records. A record can belong to several possible bundles, so its chance of appearing is the sum of those bundles' selection probabilities. The chance of one bundle is not automatically the chance of each record inside it. Looking only at the winning bundle hides the rest of the lottery.

Worked example

Consider a fictional four-record population, A, B, C and D, with values two, four, six and eight. Their true total is twenty. A sampling design selects exactly one of three pairs: AB with probability 1/2, AC with probability 1/4, or BD with probability 1/4. No other pair is possible.

A appears in AB or AC, giving inclusion probability 3/4. B similarly has probability 3/4. C and D each appear in only one possible pair and have probability 1/4. The four inclusion probabilities sum to two, matching the fixed number of distinct records selected. They need not sum to one because several records enter each sample.

The inverse-probability weights are 4/3 for A and B and four for C and D. If AB is selected, the estimated total is 2 × 4/3 + 4 × 4/3 = 8. If AC is selected, it is 2 × 4/3 + 6 × 4 = 80/3. If BD is selected, it is 4 × 4/3 + 8 × 4 = 112/3.

The design-weighted average of those estimates is (1/2) × 8 + (1/4) × 80/3 + (1/4) × 112/3 = 20. This confirms unbiasedness by enumerating all samples. Individual sample estimates need not equal the true total.

By contrast, expanding each unweighted sample total by 4/2 gives estimates 12, 16 and 24. Their design expectation is 16, below the true total. Equal expansion ignores the different chances of inclusion.

The joint inclusion probability for A and B is 1/2, not the product (3/4) × (3/4) = 9/16. C and D are never sampled together, so their joint inclusion probability is zero. Individual inclusion probabilities do not determine every aspect of the sampling design.

Illustrative case study

Fictional Willow Audits uses the four-record example to review its weighting code. The first version gives every sampled record a weight of two because half the population is sampled. It consequently produces the biased average of sixteen across the three possible outcomes.

The team replaces that rule with inverse inclusion probabilities and verifies the expectation of twenty. It also notices that the design never observes some pairs together. This matters for estimating uncertainty: unbiasedness of the total estimator does not automatically supply an unbiased variance estimator from the observed sample.

Willow records the design rather than retaining only the final selected IDs. This invented example checks calculations, not real audit performance.

Common misconceptions and mistakes

“Every probability sample gives everyone the same chance.” Known unequal probabilities can be intentional.

“Inclusion probabilities always sum to one.” Their sum equals the expected number of distinct included units. The example's sum is two.

“Per-draw probability and inclusion probability are interchangeable.” If a unit has probability 0.20 on each of two independent draws with replacement, its probability of appearing at least once is 1 − 0.80² = 0.36. Its expected number of appearances is 0.40. These are different quantities.

“A positive individual probability guarantees every pair can occur.” C and D in the example provide a counterexample.

Business use

Store selection-design information alongside sampled data. Document the target population, eligible frame, sampling stages, replacement rule and inclusion probabilities used for estimation. Keep sampling weights distinguishable from later nonresponse or calibration adjustments.

When evaluating an AI system on an oversampled subset, define whether the desired result describes that subset or a broader population. Weighting needs to match that target. A correct probability calculation cannot recover information about an excluded population group whose inclusion probability is zero, nor can it establish the accuracy of incorrectly labeled observations.

Check your understanding

Why is A's inclusion probability 3/4?

Answer: It belongs to two possible samples with probabilities 1/2 and 1/4.

Why is C's weight four?

Answer: Its inclusion probability is 1/4.

Why does the equal-weight estimator fail here?

Answer: It ignores deliberate unequal selection chances.

Related terms

Sampling design
Horvitz - Thompson estimator
Nonresponse
Inclusion probability
design weight
joint inclusion probability
probability sampling

Learn next

Change the three pair-selection probabilities while keeping them positive and summing to one. Recalculate the individual probabilities, weights and expected total, then examine which pairs remain impossible.

Sources

Peter M. Aronow and Cyrus Samii, Variance estimation for the Horvitz–Thompson estimator, Section 2, Survey Methodology, Statistics Canada: https://www150.statcan.gc.ca/n1/pub/12-001-x/2013001/article/11831/section2-eng.htm

Previous: Sampling fraction Next: Sampling rate