Can harm occur without denying someone a resource?

Yes. Depictions can affect dignity, understanding, and social standing, and can influence later treatment.

Does equal visibility eliminate representational harm?

No. People can appear equally often but be described in unequal or degrading ways.

Is every mention of a social difference harmful?

No. Meaning depends on accuracy, purpose, context, and how the difference is framed.

AI Glossary / Governance, ethics and safety / Intermediate

Representational harm

Definition

Representational harm is harm arising from how an AI system depicts, describes, categorizes, or makes people and social groups visible. Outputs can reinforce stereotypes, demean people, erase important identities or experiences, or present unequal social roles as natural. The concern is not limited to immediate denial of a resource: repeated representations can shape understanding, expectations, and social standing.

What it means

A system can cause harm through a description even when it does not decide who receives a job, loan, or service. For example, generated stories may repeatedly portray one group only in subordinate roles, or image descriptions may impose unsupported identity labels. Representational harms differ from allocational harms involving opportunities and resources, but the two can interact. Portrayals can influence later decisions, and allocation patterns can shape what appears in data. The term should name a concrete mechanism and affected people rather than serve as an unexplained synonym for any undesirable output.

How it works

Harm can arise from training material, labeling categories, generation objectives, retrieval choices, interface design, or the context in which outputs are interpreted. Measurement requires deciding what the output communicates and why that communication matters. Counting words or comparing group frequencies may reveal a pattern, but the same metric can relate to several different harms. A balanced count can coexist with degrading descriptions, while a factual historical account may legitimately have unequal counts. Appropriate evaluation combines clear definitions, contextual judgment, relevant expertise, and evidence from affected people. No single score captures every representational concern.

Three practical examples

A career-story generator repeatedly depicts certain social groups as assistants while depicting others as leaders, despite neutral requests.
An image-captioning system assigns identity labels that cannot be reliably inferred from appearance and that conflict with a person's self-description.
A search or summarization service routinely leaves out a community's contributions when answering broad questions about a field. These hypothetical examples illustrate stereotyping, imposed classification, and erasure without claiming that every omission has the same meaning or severity.

Think of it this way

Think of a public exhibition whose captions and selection of portraits shape visitors' understanding of who matters. Harm can come from insulting captions, misleading patterns, or systematic absence. The analogy is limited because AI can generate many different outputs interactively and distribute them at scale. Its effects depend on the prompt, audience, language, and surrounding use rather than one fixed display.

Worked example

Suppose a fictional evaluation generates 100 short workplace stories using a defined prompt set. Reviewers find that 50 stories depict Group A and 50 depict Group B, so a simple appearance-count metric shows equal representation. They then code roles: 40 of A's characters are leaders, compared with 10 of B's. Leadership portrayal rates are 80% and 20% within the depicted groups. This pattern motivates investigation of stereotyping, but the counts alone do not establish the full harm or its real-world prevalence. Reviewers must examine prompt design, context, coding reliability, and the actual descriptions. Rebalancing leadership counts would also not rule out derogatory language or erasure of other groups. The example shows why the harm definition must guide the measurement.

Illustrative case study

Fictional case: Silverleaf Media uses an AI tool to draft educational examples. A review panel notices that the examples include a broad range of names but repeatedly assign narrow roles to some communities. Editors develop a more precise review rubric, consult relevant readers, and revise both prompts and review practices. They check whether the changes introduce new distortions. This invented case illustrates that superficial diversity is not equivalent to respectful and accurate representation, and that mitigation requires attention to meaning as well as counts.

Common misconceptions and mistakes

Representational harm is not merely a matter of whether an output contains a banned word. A seemingly polite description can reinforce a harmful stereotype, while quoting harmful language in a critical historical discussion has a different context. Statistical parity is not a complete remedy. Nor is deleting all references to an identity automatically harmless: it can erase relevant experience or make a service less useful. An accurate statement about one person should not be generalized into a claim about an entire group. Evaluation needs to distinguish depiction, inference, and endorsement.

Business use

Define the representational concerns relevant to the product and its audiences. Use examples and annotation guidance that explain why a pattern matters, and record uncertainty or disagreement rather than forcing false consensus. Test across languages, contexts, and meaningful intersections where the system is used. Combine quantitative screening with qualitative review and feedback channels. Avoid claiming that a benchmark improvement proves the absence of harm. When mitigating, check for tradeoffs such as overbroad suppression, inaccurate rewriting, or loss of useful self-description. Give editors or users practical ways to correct problematic outputs.

Check your understanding

A generator includes equal numbers of two groups but consistently portrays one as less capable. Does its equal-count metric settle the fairness question?

Answer: No. The metric measures visibility, while the concern involves the meaning and roles in the portrayals. A more appropriate evaluation is needed.

Related terms

Representation Bias
stereotyping
erasure
allocational harm
social bias
image tagging
harmful language
measurement validity
fairness evaluation

Learn next

Pick one hypothetical output pattern and state the specific representational harm it could cause. Identify who is affected, how the output communicates the problem, and what a proposed metric would miss. Add a qualitative check before drawing a conclusion.

Sources

Katzman and colleagues, original taxonomy and measurement analysis: https://arxiv.org/html/2305.01776v1 Harvey and colleagues, practitioner needs in measuring representational harms: https://aclanthology.org/2025.findings-acl.947/

Previous: Representational frame problem Next: Representational similarity analysis