AI Glossary / Governance, ethics and safety / Intermediate
Reference class
Definition
A reference class is a group of cases used as the comparison basis for a judgment, estimate, or probability about a particular case. The group is chosen because its members share relevant characteristics with the case being considered. Different plausible reference classes can produce different estimates, so choosing and explaining the comparison group is part of the reasoning, not a minor administrative detail.
What it means
Suppose a team wants to estimate whether a new AI project will finish on time. It could compare all software projects, AI projects, projects of similar size, or projects using the same integration approach. Each group may have a different history. The question is which similarities matter for the outcome and whether enough trustworthy evidence exists. A reference class is not simply the model's predicted class, nor is it necessarily a legally protected class. It is the set of comparable cases from which a relevant pattern or base rate is drawn.
How it works
Define the outcome and target case, identify candidate comparison groups, and gather consistent observations about their outcomes. Examine how the cases were selected and whether important failures or incomplete projects are missing. Estimate the relevant distribution or frequency, with uncertainty, and assess how well it transfers to the target. A narrow class may be more similar but contain too little evidence; a broad class may have many observations but mix unlike situations. The reference-class problem is that one case belongs to many groups, and probability rules alone do not always determine which grouping should guide a particular judgment.
Three practical examples
- A planning team compares a proposed AI deployment with earlier deployments that required similar data integration and review workflows.
- An evaluator compares a model's error rate on documents from a specified language and capture process, rather than using an unrelated overall rate.
- A reliability analyst examines failures among devices operated under comparable conditions when estimating expectations for a new installation. These examples depend on justified comparability, not merely on finding a convenient historical number.
Think of it this way
Think of asking people about the length of a journey. Advice from travelers on the same route and transport mode is usually more relevant than an average across all journeys, but a handful of unusual trips may still mislead. The analogy is limited because deciding what counts as the same route is itself part of the problem. Relevant similarity is defined by the outcome and evidence, not by surface resemblance alone.
Worked example
Suppose an organization has records of 100 completed technology projects, 30 of which exceeded their planned duration. Its broad observed overrun rate is 30%. Within those records, 20 projects involved integrating three or more legacy systems, and 10 of those overran, giving 50%. A new project also has several legacy integrations. The narrower reference class may be more relevant, but it has only 20 observations and may differ in staffing, size, or time period. The two rates are conditional on different group definitions; neither arithmetic result is wrong merely because they differ. The team should explain its selection and uncertainty, avoid claiming an exact individual probability, and check whether canceled or still-running projects were excluded in a way that distorts the comparison.
Illustrative case study
Fictional case: Oakbridge Analytics forecasts a launch using the team's fastest prior prototype as its main analogy. A reviewer asks for a reference class that includes comparable projects with security review, data cleaning, and user training. The broader evidence reveals stages omitted from the initial estimate. The team revises its plan and documents why those cases are relevant. This invented case illustrates how a reference class can challenge an overly optimistic story without guaranteeing that the revised forecast will be correct.
Common misconceptions and mistakes
A reference class is not automatically objective because it contains numbers. Selecting only successful or familiar cases can make the comparison misleading. A very specific class is not always better if its sample is tiny or selected after seeing outcomes. A large class is not always better if it combines incompatible contexts. The average outcome does not describe every member, and applying a group frequency to an individual requires judgment about the available information. Reference-class forecasting complements case-specific analysis; it does not require ignoring every distinctive feature of the new situation.
Business use
Use reference classes to make assumptions visible in planning, risk assessment, and model evaluation. Define inclusion criteria before selecting favorable examples, and retain information about failures and missing outcomes. Compare several defensible classes to see how sensitive the estimate is to the choice. Report distributions and uncertainty rather than only a single average. Explain relevant changes since the historical cases occurred, such as new technology or operating procedures. Where evidence is weak, acknowledge the limit and collect better comparables instead of treating a borrowed base rate as a precise prediction.
People also ask
Can one case belong to several reference classes?
Yes. That is why the choice needs justification.
Is the smallest reference class always best?
No. Greater similarity can come at the cost of too little or unrepresentative evidence.
Does a group frequency determine an individual's outcome?
No. It provides statistical evidence, not certainty about what will happen in one case.
Check your understanding
A new project belongs to a broad group with a 30% overrun rate and a narrower group with a 50% rate. Must one estimate be a calculation error?
Answer: No. The groups have different definitions and can legitimately have different observed frequencies. Relevance, sample quality, and uncertainty determine how the evidence should be used.
Related terms
- Base rate
- Conditional probability
- Generalization
- Selection bias
- Uncertainty
- reference-class forecasting
- comparison group
- planning fallacy
- outside view
Learn next
Choose a hypothetical forecast and define three plausible reference classes. For each, state why membership may matter, how many usable cases exist, and what important cases could be missing. Explain which estimate you would use and how the alternatives change your confidence.
More in Governance, ethics and safety
- Ranking Bias - Ranking bias is a tendency to treat highly ranked results as especially relevant or important partly because of their position.
- Rectification - Rectification, in personal-data governance, means correcting inaccurate personal information or completing information that is incomplete for its relevant...
- Red teaming - Red teaming is a structured, authorized effort to find weaknesses, harmful behaviors, or exploitable failure modes in an AI system.
- Referential transparency - Referential transparency concerns whether an expression can be replaced by an equivalent expression or value without changing the meaning or relevant...
- Regularized empirical risk minimization - Regularized empirical risk minimization, or RERM, is a model-training principle that balances average loss on training data with a penalty expressing a...
- Relative risk or risk ratio - Relative risk, or risk ratio, compares the probability of a specified event in one group or condition with its probability in another.
Sources
Halpern, original discussion of reference-class difficulties: https://www.cs.cornell.edu/info/people/halpern/papers/kyburg_response.pdf Flyvbjerg, From Nobel Prize to Project Management: Getting Risks Right: https://arxiv.org/abs/1302.3642 Kyburg, original chapter on probability and reference-class choice: https://conservancy.umn.edu/bitstreams/2e3209f0-bd7b-480f-bceb-4edbf4e56755/download
Previous: Redundant path Next: Reference class problem