Is systemic the same as systematic?

Systematic generally means patterned rather than random; systemic specifically points to the operation of a wider system or institution.

Can AI create new systemic problems?

Its deployment can change processes and distribute benefits or burdens in new ways.

Is systemic bias only a dataset issue?

No. It can arise in objectives, institutional rules, and the use of outputs as well.

AI Glossary / Governance, ethics and safety / Advanced

Systemic bias

Definition

Systemic bias is a pattern of advantage or disadvantage arising from the procedures, norms, and practices of institutions or social systems. In AI, it can influence the problem being solved, the records available for training, the meaning of labels, and how outputs affect people. It does not require an individual developer to intend discrimination. NIST distinguishes systemic bias from statistical and computational bias and from human cognitive bias, while recognizing that they interact.

What it means

An AI system operates inside an existing arrangement of rules and opportunities. If some people have historically had less access to a service, their lower recorded use may reflect that barrier rather than lower need. A model trained to predict recorded use can reproduce the pattern accurately while remaining unsuitable for allocating support according to need.
Systemic bias is therefore not simply a coding error or an unbalanced dataset. Even an accurate record of existing practice can preserve an inequitable process.

How it works

Institutional practices influence who enters a process, what is recorded, and which outcomes are treated as desirable. Those choices become features, targets, and evaluation criteria. Model outputs then enter workflows that can reinforce the original pattern, particularly if the resulting decisions generate the next training dataset.
Suresh and Guttag's framework distinguishes sources of harm across the machine learning lifecycle, including historical, measurement, and deployment problems.
Investigation should connect observed differences to specific mechanisms. Examine access rules, label construction, decision authority, and routes for correction. Outcome differences can signal a problem, but a single group comparison does not by itself establish the cause or identify an effective remedy.

Three practical examples

A training recommendation tool treats previous attendance as evidence of interest. If sessions were consistently scheduled when some staff could not attend, the recommendation process may carry that scheduling disadvantage forward.
An AI support service accepts only one communication format. People who cannot use that format receive less help, even if the model performs equally well on every request it actually receives.
A model learns from historical promotion ratings. If those ratings reward access to opportunities that were unevenly distributed, reproducing the ratings faithfully does not establish that the new selection process is fair.

Think of it this way

Consider a timetable that repeatedly schedules a shared activity when one group cannot participate. Counting attendance perfectly will not reveal equal opportunity, and a prediction of future attendance will not repair the timetable. The measurement, prediction, and underlying arrangement need to be examined together.

Worked example

Imagine a fictional organization with two equally sized staff groups, A and B, each containing 100 people. For this example, all 200 want the same training and meet its eligibility requirements.
Because of the existing session schedule, 80 people in A and 20 in B attended last year. A proposed recommendation rule gives priority only to previous attendees. It therefore recommends 80% of A and 20% of B, a difference of 60 percentage points.
Suppose a model reproduces that rule perfectly. Its accuracy against the historical attendance label is 100%, yet it has learned attendance under the old schedule rather than the stated objective of identifying interested, eligible staff. Perfect label prediction cannot resolve that mismatch.
Now imagine sessions are redesigned and attendance is independently reassessed. If 70 people from each group attend, the new rates are both 70%. That outcome would be consistent with the scheduling explanation in this invented scenario, but an uncontrolled before-and-after comparison in a real organization would require further investigation of other changes.
The numbers illustrate a mechanism, not a universal fairness formula. In practice, interest, eligibility, scheduling constraints, and the purpose of the recommendation need evidence rather than assumptions.

Illustrative case study

Fictional company AlderServices introduces an AI tool to prioritize customer support improvements. It trains on completed online requests because these records are easy to analyze. Customers who rely on an inaccessible submission route rarely appear in the dataset.
A review initially considers balancing the recorded customer groups. Interviews and process mapping reveal a more basic issue: some people cannot submit a request at all. Rebalancing existing records would not represent those absent experiences reliably.
The company improves the entry process, provides an alternative route, and separately evaluates submission access and resolution quality. This fictional case illustrates an institutional mechanism that extends beyond the model; it does not claim that one redesign removes every source of disadvantage.

Common misconceptions and mistakes

“No one intended harm, so there is no systemic bias.” Institutional routines can produce disadvantage without conscious prejudice.
“Representative data guarantee fairness.” Data can accurately represent an unfair arrangement.
“Removing group attributes removes the problem.” Other information or the surrounding process can preserve the same mechanism.
“Every group difference proves systemic bias.” Differences need investigation, relevant context, and a clear account of the decision being evaluated.
“One numerical fairness metric settles the issue.” A metric captures a particular property, not every aspect of access, treatment, or impact.

Business use

Use this concept when defining an AI project's purpose and reviewing its actual effects. Ask whether the target reflects the desired outcome or merely the institution's past behavior. Include people familiar with how the process works for those it serves, especially experiences absent from routine reports.
Assign responsibility for changes outside the model, such as access channels or review procedures. Record the suspected mechanism, proposed response, and evidence used to assess improvement. Treat technical adjustments as part of a wider process review rather than as automatic proof of resolution.

Check your understanding

Why is perfect attendance prediction insufficient in the worked example?

Answer: Attendance under an unequal schedule is not the same target as interest and eligibility.

Why might collecting more old records fail to help?

Answer: It could preserve the same access and labeling mechanism.

What should follow an observed disparity?

Answer: Investigation of the process and evidence about causes, consequences, and possible remedies.

Related terms

Historical bias
Institutional Bias
Measurement Bias
Feedback loop
proxy variable
fairness
sociotechnical system
accessibility

Learn next

Map one AI-supported decision from access to final outcome. Identify who can enter, what the target measures, and which institutional rule could affect the evidence. Propose a way to investigate that mechanism.

Sources

NIST, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, Section 2.1.2: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf Suresh and Guttag, A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle: https://arxiv.org/abs/1901.10002

Previous: Systematic resampling Next: t-Distributed Stochastic Neighbor embedding