Is the sample size fixed?

Usually not in the event-target version.

Does success mean good performance?

No; it can mean a counted failure.

Can the target take a long time?

Yes, particularly for rare events.

AI Glossary / Statistics and probability / Advanced

Inverse sampling

Definition

Inverse sampling is a sampling design in which observations continue until a specified condition based on the observed results is met. A common version stops after a predetermined number of successes, leaving the total number of observations random. Here, success simply means the event being counted; it need not represent a desirable outcome.

What it means

An ordinary fixed-size study might inspect 100 items and count defects. An inverse-sampling study might instead inspect items until it finds ten defects and record how many inspections were necessary. The fixed and random quantities have exchanged roles.
The stopping rule is part of the statistical design. A dataset containing ten defects among 100 inspections does not reveal whether 100 inspections were planned or whether the hundredth inspection produced the tenth defect. That distinction affects the sampling distribution used for inference.

How it works

Specify the counted event, observation process, stopping target, and any operational cap before collection. Preserve the observations that fail to meet the counted condition as well as those that meet it.
For independent Bernoulli trials with constant success probability p, let r be the target number of successes and N the total trials needed. N follows the total-trials version of a negative binomial distribution. Some references instead count failures, X = N minus r, so their formulas use a different variable.
Under this model, the maximum-likelihood estimate r/N is biased. For r greater than one, (r minus 1)/(N minus 1) is an unbiased estimator of p. Those properties depend on the stated design and assumptions; they are not universal corrections for any stopped dataset.

Three practical examples

An AI evaluation team samples independent simulated requests until it observes a specified number of a defined failure type, retaining the total number tested.
A survey researcher screens randomly selected records until enough members of a target group are identified, using an analysis appropriate to the selection design.
A quality analyst plans inspection around a target number of defects, while separately budgeting for uncertain inspection effort and possible noncompletion.

Think of it this way

Imagine counting passing vehicles until the third blue vehicle appears. Three is fixed, but the number of vehicles you must watch varies. The vehicles of other colors are essential information about how frequently blue vehicles occur.

Worked example

Consider an original simplified evaluation model. Each trial independently produces the counted event with probability p = 0.25. The team stops at its third event, so r = 3. This known probability is used to demonstrate the design, not inferred from the observations below.
Suppose the events occur on trials two, six, and ten. The experiment stops at N = 10, after three successes and seven failures. The maximum-likelihood estimate is 3/10 = 0.30. The unbiased estimator for this design is 2/9, approximately 0.2222.
Neither estimate must equal the generating probability in an individual run. Unbiasedness concerns the average over repetitions of the complete sampling procedure, not whether a particular estimate is closest to the truth.
For the experiment to stop on trial five, the first four trials must contain exactly two successes and trial five must be a success. There are six possible placements of those first two successes. Each complete sequence contains three successes and two failures.
Thus P(N = 5) = 6 times (1/4) cubed times (3/4) squared = 27/512, approximately 5.27 percent. Counting any three successes among all five trials would incorrectly include sequences that should already have stopped earlier.
The possible stopping times begin at three. Their probabilities at three and four are respectively 1/64 and 9/256. Adding those to the probability at five gives P(N at most 5) = 53/512, approximately 10.35 percent.
Now suppose an operational cap permits only five trials. The target will remain unmet with probability 459/512, approximately 89.65 percent. Reporting only the runs that finish would select an unusual subset of the planned experiments. Capped runs need to remain visible, and the analysis must accommodate the cap.
Without that cap, the expected total is r/p = 3/0.25 = 12 trials. The variance is r times (1 minus p) divided by p squared = 36, giving a standard deviation of six trials. An expected workload of twelve therefore does not guarantee completion within twelve.
At four minutes per trial, expected trial time is 48 minutes, excluding setup and review. If the event probability were 0.05 instead, the expected total would rise to 60 trials, or four hours of trial time. The same target can have very different operational costs.
These calculations assume independent trials with an unchanged probability. Repeatedly testing nearly identical requests, changing the model halfway through, or selectively choosing promising cases would require a different analysis.

Illustrative case study

Fictional company Birch Evaluation wants several examples of a particular assistant failure for diagnosis. It records every sampled trial and stops when the third qualifying failure occurs.
A reviewer notices that a five-trial budget often prevents completion. The team separates completed and capped runs, retains their full records, and revises its workload planning. It also distinguishes collecting diagnostic examples from estimating failure prevalence. This fictional scenario illustrates design choices, not measured product performance.

Common misconceptions and mistakes

Inverse sampling is not inverse-transform sampling, which generates random values using a distribution's inverse cumulative function.
It is not automatically inverse-probability weighting.
Keeping only successes loses the denominator needed to understand event frequency.
Stopping after a target number of events does not make trials independent or eliminate selection bias.
A finite population sampled without replacement needs its own sampling distribution; the independent-trial formula should not be applied mechanically.

Business use

Use this design when obtaining a specified number of qualifying observations matters and variable effort is acceptable. Record stopping rules alongside results so later analysts can reconstruct the experiment. Plan staffing around workload uncertainty and define how incomplete runs will be handled before collection begins.

Check your understanding

What stops the example experiment?

Answer: Its third qualifying event.

What is random?

Answer: The total number of trials.

Is twelve trials a guaranteed completion time?

Answer: No; it is the model's expectation.

Related terms

Stopping rule
Bernoulli trial
Censoring
Inverse transform sampling
Sequential sampling
negative binomial distribution
survey sampling

Learn next

Compare fixed-size and event-target experiments, then study inference under negative binomial sampling and designs with operational stopping caps.

Sources

OECD, Glossary of Statistical Terms, inverse sampling: https://www.oecd.org/content/dam/oecd/en/publications/reports/2008/09/oecd-glossary-of-statistical-terms_g1gh9ad7/9789264055087-en.pdf . Statistics Canada, Unequal probability inverse sampling, Section 3: https://www150.statcan.gc.ca/n1/pub/12-001-x/2016002/article/14660/03-eng.htm . Numerical examples and company scenario are original illustrations.

Previous: Inverse reinforcement learning Next: Inverse transform sampling