AI Glossary / Governance, ethics and safety / Intermediate

Disparate impact

Definition

Disparate impact refers to a practice producing disproportionately unfavorable outcomes for a group, even when the practice appears neutral on its face. In AI discussions, it often concerns differences in access, selection, or burdens across groups. The term also has specific legal meanings that depend on the jurisdiction and setting; a statistical disparity alone does not settle those legal questions.

What it means

A system need not explicitly use a protected characteristic for its outcomes to differ across groups. Other inputs, selection rules, or stages of a workflow can produce uneven effects. For example, an assessment based on a narrowly defined experience pattern may exclude people whose relevant experience is recorded differently. A useful review asks what outcome is distributed, who is affected, and which practice contributes to the pattern. It also distinguishes effects from intent: unequal outcomes do not by themselves establish deliberate discrimination, while lack of discriminatory intent does not make outcome analysis unnecessary.

How it works

Analysts first identify a meaningful decision and comparison population. For a favorable binary decision, they can calculate each group's selection rate and compare rates through a difference or ratio. They then examine uncertainty, sample size, data quality, relevant process stages, and possible mechanisms. This is only part of an assessment. The purpose and relevance of the procedure, the consequences of errors, and available alternatives also matter. In U.S. employment guidance, the four-fifths rule is a screening rule of thumb for selection-rate differences, not a universal definition of fairness or an automatic determination of legal liability.

Three practical examples

A business reviews whether an AI-assisted shortlisting stage selects applicants from different groups at substantially different rates, using an appropriate applicant population.
A service provider examines whether an automated verification step disproportionately prevents some customers from completing registration.
An education platform investigates whether a recommendation rule repeatedly limits access to advanced material for a particular learner group. The relevant domain rules and evidence differ across these settings.

Think of it this way

Think of an entrance with a step that appears identical for everyone but creates different access barriers for different visitors. Uniform design does not ensure uniform practical effect. The analogy is limited because statistical selection patterns involve uncertainty, multiple causes, and legal context. It helps separate the written rule from the consequences experienced by people who encounter it.

Worked example

Suppose a fictional selection process considers 200 applicants in Group A and selects eighty. Its selection rate is 80/200 = 40 percent. It considers one hundred applicants in Group B and selects twenty, giving 20/100 = 20 percent. Group B's rate divided by Group A's rate is 0.20/0.40 = 0.50, or 50 percent. The rate difference is twenty percentage points. This ratio would fall below the 80 percent screening threshold associated with the U.S. employment four-fifths rule. That observation calls for investigation; it does not independently establish unlawful discrimination or identify the cause. The comparison needs a suitable applicant pool, accurate group and outcome data, the applicable legal framework, and examination of the selection procedure. The raw selected counts, eighty and twenty, are not enough because the group denominators differ.

Illustrative case study

Fictional case: Hawthorn Services reviews an AI-assisted application workflow and finds a selection-rate gap. Looking at each stage reveals that a document-format check rejects many otherwise complete applications before substantive review. The team investigates accessibility, the relevance of the format requirement, and an alternative verification process. It evaluates the revised process with appropriate expertise rather than declaring success because a single ratio moved closer to one. This invented case shows how outcome analysis can lead to a concrete process investigation without attributing a real result to a company.

Common misconceptions and mistakes

Removing a sensitive feature from model inputs does not guarantee equal outcomes, because other features and surrounding procedures still matter. Passing a four-fifths screen is not a blanket fairness certificate, and falling below it is not a complete legal finding. Equal selection rates can coexist with unequal errors or inaccessible procedures. Another mistake is comparing only final outcomes when earlier stages already exclude people. Conversely, slicing data into very small groups without considering uncertainty can produce unstable conclusions. Statistical evidence should be interpreted with the actual decision process and affected population.

Business use

Define favorable and unfavorable outcomes clearly, retain meaningful denominators, and examine important stages of the workflow. Work with relevant legal, statistical, domain, and accessibility expertise where the decision warrants it. Protect sensitive information used for assessment and establish an appropriate basis for collecting or using it. Investigate procedures that may explain gaps, including data requirements and human overrides. Consider alternatives that meet legitimate objectives with fewer harmful effects. Document what the evidence supports and what remains uncertain; do not turn a single metric into the entire governance decision.

Check your understanding

Two groups each have forty selected applicants, but one group had one hundred applicants and the other had two hundred. Are their selection rates equal?

Answer: No. The rates are 40 percent and 20 percent. Equal selected counts can conceal unequal rates when denominators differ.

Related terms

Demographic parity
Disparate treatment
Subgroup analysis
Algorithmic accountability
adverse impact
selection rate
fairness assessment
proxy variable
accessibility

Learn next

Draw the stages of a selection workflow and record the number entering and passing each stage for relevant groups. Calculate rates with clearly defined denominators, then investigate where a meaningful difference first appears and what evidence could explain it.

Sources

U.S. EEOC, Employment Tests and Selection Procedures: https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures U.S. EEOC, Questions and Answers on Uniform Guidelines: https://www.eeoc.gov/es/node/130157 Fairlearn, Performing a Fairness Assessment: https://fairlearn.org/main/user_guide/assessment/perform_fairness_assessment.html

Previous: Disordinal interaction Next: Disparate treatment