Is there one best fairness metric?

No. The appropriate measure depends on the decision, harm, population, and assumptions. Several complementary measures may be needed.

Does equal accuracy imply fairness?

No. Groups can have the same accuracy while experiencing different error types or downstream consequences.

Does a disparity prove intentional discrimination?

No. A metric describes a pattern under its definitions. Explaining causes or assessing legal responsibility requires additional evidence and context.

AI Glossary / Governance, ethics and safety / Intermediate

Fairness metric

Definition

A fairness metric is a quantitative measure used to examine whether an AI system meets a specified fairness criterion. It may compare selection rates, error rates, or other outcomes across groups, or assess consistency between relevantly similar individuals. A metric makes one aspect of fairness measurable; it does not provide a complete judgment that a system or the surrounding decision process is fair.

What it means

Fairness can mean different things in different settings. A team might ask whether groups receive comparable access to a service, whether qualified people are overlooked at similar rates, or whether similar individuals receive similar treatment.

Those questions require different measurements. A selection-rate comparison concerns who receives an outcome, while an error-rate comparison also depends on a reference label describing what should have happened. If the reference labels are poor, the second comparison may inherit their problems.

Choosing a metric therefore requires stating the concern, affected population, decision, and available evidence. Starting with a convenient number and calling it “the fairness score” hides these choices rather than resolving them.

How it works

Define the outcome and the fairness question before calculating. Identify the population, relevant groups or similarity criteria, and the observations needed for the chosen measure. Document how labels and group information were obtained and what limitations they have.

Compute the measure with clear denominators. For example, demographic-parity assessments compare selection rates, while equalized-odds assessments compare both true-positive and false-positive rates across groups. Different criteria can reveal different patterns and may not be jointly achievable under all conditions.

Interpret the results alongside sample sizes, uncertainty, data quality, and the actual decision workflow. Examine relevant subgroups where the data support it, and investigate observed disparities. A numerical gap is evidence for inquiry, not by itself a complete explanation of its cause or an automatic instruction for a particular remedy.

Three practical examples

1. A service-access review compares how often eligible requests from different language groups reach a successful completion, using a defined eligibility reference.

2. A classification team compares false-positive and false-negative patterns across relevant groups instead of relying only on an overall accuracy figure.

3. An allocation review compares selection rates and then examines whether the selected measure captures the harm people actually experience in that process.

Think of it this way

Think of a fairness metric as one instrument on an inspection panel. A temperature gauge and a pressure gauge answer different questions, and a normal reading on one does not certify the entire machine.

The analogy is limited because fairness includes social judgments about what should be measured and what outcomes are acceptable. There is no single physical quantity called fairness that every instrument estimates in the same way. Human interpretation and context remain part of the assessment.

Worked example

Suppose a fictional service selects 60 of 100 requests from group A and 30 of 100 from group B. Selection rates are 60% and 30%. The absolute difference is 30 percentage points, while the lower-to-higher rate ratio is 0.30/0.60 = 0.50.

Now suppose 80 requests in A and 40 in B meet a separately reviewed eligibility criterion, and all selected requests are eligible. The true-positive rates are 60/80 = 75% and 30/40 = 75%.

Thus selection rates differ while the true-positive rates match in this invented dataset. This is not a verdict that the process is fair or unfair overall. It shows why the question and denominator matter. The eligibility criterion itself, false-positive rates, excluded applicants, and practical consequences still require attention. These point estimates also omit statistical uncertainty.

Illustrative case study

Fictional case: Openfield Services introduces an automated request-routing system. A dashboard shows similar overall accuracy across two groups, but user reports suggest one group experiences more unnecessary escalations.

The team adds a measure for that error type, checks the reference labels, and reviews the resulting cases with staff familiar with the service. It also separates model recommendations from the final actions taken by reviewers.

This fictional investigation treats metrics as diagnostic tools. It does not assume a matching aggregate score establishes equal experience or that the model alone caused every observed difference.

Common misconceptions and mistakes

Do not present one metric as a universal certificate of fairness. Different measures operationalize different concerns and can conflict.

Do not report gaps without group counts and a clear definition of the denominator. Small samples and missing observations can make comparisons unstable or misleading.

Avoid assuming equal model error rates guarantee equal downstream outcomes. Human review, service availability, and other process stages can change what people experience. Conversely, an observed disparity does not establish its cause without further investigation. Keep numerical assessment separate from unsupported causal or legal conclusions.

Business use

Fairness metrics help teams make concerns explicit, monitor changes, and identify where investigation is needed. They are most useful when tied to a concrete decision and an accountable review process.

Document why each metric was selected, what it omits, and how results influence decisions. Compare new model versions using the same definitions while checking whether the population or workflow has changed. Combine quantitative findings with evidence from affected users and domain specialists, especially where existing labels do not fully describe the consequences.

Check your understanding

Two groups have equal true-positive rates but different selection rates. Must one of the calculations be wrong?

Answer: No. They use different denominators and measure different properties. Check the definitions and data before treating the results as contradictory.

Related terms

Learn next

Learn demographic parity and equalized odds next, calculating each from the same small confusion tables. Then study metric incompatibility and construct validity to understand why a mathematically correct number can still be an incomplete or unsuitable assessment of the underlying concern.

Sources

Fairlearn primary documentation, common fairness metrics: https://fairlearn.org/main/user_guide/assessment/common_fairness_metrics.html

Google Machine Learning Glossary, fairness metrics and related definitions: https://developers.google.com/machine-learning/glossary

Primary documentation was read for metric definitions and their contextual limitations. The request counts, review guidance, and Openfield Services case are original teaching illustrations. No numerical threshold here is presented as a legal compliance test.

Previous: Fairness constraint Next: False discovery rate