Can it still be useful?

Yes, especially for purposeful investigation.

Does weighting automatically remove bias?

No; unmeasured or within-category selection can remain.

AI Glossary / Statistics and probability / Advanced

Judgment sample

Definition

A judgment sample is a non-probability sample whose members are deliberately selected using a researcher's or expert's assessment of their relevance or usefulness. It is also called purposive sampling. Selection follows a reasoned choice rather than a probability design with known selection chances. A carefully chosen sample can be informative without supporting representative population estimates.

What it means

A team may deliberately choose difficult AI requests, experienced users, unusual incidents, or contrasting cases because those observations are useful for its question. This can be an appropriate way to investigate mechanisms, test a questionnaire, or discover failure modes.
The same selection can distort an estimate of how common a behavior is. Choosing cases because they are revealing, problematic, or accessible changes what is observed. Expert knowledge helps define the investigation but does not automatically make the selected cases representative of everyone else.

How it works

Define the purpose, relevant population, and criteria for inclusion. Explain who selects cases, what information guides the selection, and which types of cases may be excluded. Record selection criteria.
Analyze findings in relation to that purpose. Describe observed cases directly and distinguish them from claims about unobserved units. Population inference from non-probability data requires additional assumptions and supporting information; it does not inherit the ordinary sampling guarantees of a random design.
Increasing the number selected does not by itself remove systematic selection bias. Likewise, matching a few known population characteristics does not establish that selection is unrelated to the outcomes within those categories.

Three practical examples

An AI evaluator selects particularly difficult prompts to expose weaknesses that might be missed in a small random sample.
A product researcher interviews users with contrasting levels of experience to investigate how they interpret a new feature.
A survey designer chooses people likely to encounter confusing wording while pilot testing a questionnaire, before a separate population study.

Think of it this way

A mechanic inspecting the parts most likely to explain a strange noise is making a purposeful selection. Those parts may be exactly right for diagnosis, but their condition does not automatically describe the average condition of every part in the vehicle.

Worked example

Consider an original population of 1,000 recorded AI requests. There are 100 requests classified as difficult and 900 classified as routine. For illustration, suppose a complete inspection would find 30 failures among the difficult requests and 18 among the routine requests.
The true overall failure proportion in this invented population is therefore (30 plus 18)/1,000 = 0.048, or 4.8 percent. The difficult-request failure rate is 30 percent, and the routine-request rate is two percent.
An expert deliberately selects 50 difficult requests and 50 routine requests for diagnosis. Among the selected difficult cases, 20 fail; among the selected routine cases, one fails. The selected sample's failure proportion is 21/100 = 21 percent.
That is a correct description of the selected cases, but it is not the population rate of 4.8 percent. The sample gives difficult requests half its weight even though they make up only one tenth of the population. It also selects a difficult subgroup with a 40 percent failure rate rather than the population subgroup's 30 percent rate.
Suppose an analyst adjusts only for the known difficult/routine proportions. Weighting the selected rates gives 0.10 times 0.40 plus 0.90 times 0.02 = 0.058, or 5.8 percent.
The adjustment removes the imbalance in the two broad category shares but leaves the within-difficult-category selection difference. It is closer to the invented truth, yet still one percentage point too high. Matching known category totals does not necessarily repair selection within categories.
Now imagine that the complete-inspection results were unavailable. The analyst knows only the population size and the 100 inspected outcomes, including 21 failures. Without assumptions about the remaining 900 requests, the total number of failures could be as low as 21 or as high as 921.
The corresponding possible population rates range from 2.1 percent to 92.1 percent. These are logical bounds from observed and unobserved binary outcomes, not a confidence interval. They demonstrate how little a selected sample alone can establish about unobserved cases.
For a separate expansion of the same invented population, suppose the team inspects all 100 difficult requests and deliberately chooses 100 routine requests, finding 30 and two failures respectively. Its combined sample rate is 32/200 = 16 percent.
Doubling the sample size has not made that unweighted rate equal the population rate. The composition still gives difficult requests half the sample weight. Weighting the two rates by the population shares now gives 4.8 percent, but the match depends on the selected routine cases having the same failure rate as all routine cases. The deliberate selection alone does not guarantee that equality.
The original 21 failures may reveal several distinct problems worth investigating. Finding a reproducible failure establishes that it can occur under the tested conditions; it does not establish how frequently it occurs across all requests.
These invented counts expose selection effects that would usually be unknown in practice.

Illustrative case study

Fictional company Maple Evaluation builds a deliberately challenging prompt set to investigate an assistant's weaknesses. It documents why each case was chosen and groups observed failures by mechanism.
The team uses those findings to improve testing and design a separate study of ordinary usage. Its report labels the challenge-set failure proportion clearly instead of presenting it as a deployment-wide rate. This fictional case illustrates appropriate use of targeted evidence.

Common misconceptions and mistakes

Expert selection does not create known sampling probabilities.
A large selected sample can retain substantial selection bias.
Balanced category counts do not prove representativeness within categories.
A sample useful for discovering problems need not estimate their prevalence.
Ordinary random-sampling margins of error should not be attached without a justified inferential basis.

Business use

Judgment samples are useful for exploration, qualitative investigation, pilot work, and targeted evaluation. Match the claims to the selection method. If a decision depends on population prevalence, plan a suitable probability sample or a defensible model-based analysis with explicit assumptions and external checks.

Check your understanding

What is the first selected-sample failure rate?

Answer: Twenty-one percent.

What is the invented population rate?

Answer: 4.8 percent.

Why does category weighting initially give 5.8 percent?

Answer: The selected difficult cases have an elevated within-category failure rate.

Related terms

Selection bias
Convenience sample
Purposive sampling
non-probability sampling
quota sampling
probability sampling
sampling frame
generalizability

Learn next

Compare purposive selection with stratified random sampling, then examine what assumptions are needed to generalize from non-probability data.

Sources

Statistics Canada, Non-probability sampling, Judgement sampling: https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch13/nonprob/5214898-eng.htm . Australian Bureau of Statistics, Basic Survey Design: Sample Design: https://www.abs.gov.au/websitedbs/d3310114.nsf/home/Basic%20Survey%20Design%20-%20Sample%20Design . Counts, weighting illustrations, and company scenario are original.

Previous: JTMS Next: Jump point search