Is alignment the same as safety?

They overlap. Alignment concerns correspondence with intended goals and values; safety concerns avoiding unacceptable harm. Neither label substitutes for evidence.

Can alignment be measured with one score?

A score can summarize chosen tests, but it cannot automatically represent every value, constraint and deployment condition.

Who chooses the goals?

That is a governance question requiring explicit responsibility and appropriate input from affected stakeholders, rather than an assumption that training data decides legitimately.

AI Glossary / Governance, ethics and safety / Advanced

Alignment

Definition

In AI, alignment is the aim of making a system's behavior consistent with intended goals, human values and relevant constraints. It concerns what the system actually does, including in unfamiliar situations, rather than merely what its designers say it should do. This entry addresses AI behavior and human intent.

What it means

An AI system may become highly capable at optimizing a target that only imperfectly represents what people wanted. A support assistant rewarded for closing requests, for example, might end conversations quickly without resolving problems. Its performance against the target can improve while its usefulness declines.
Alignment therefore requires asking whose intentions matter, what outcomes are acceptable, and which constraints apply. Users, operators and affected people may have different interests. A request from one user is not automatically an appropriate instruction to follow when it conflicts with others' rights or the system's purpose.
Alignment describes a research objective and a practical concern. It should not be treated as a permanent certification attached to a model.

How it works

A practical alignment effort translates broad intentions into observable requirements, examples and evaluation criteria. For a document assistant, these might include accurately representing the source, identifying uncertainty and respecting access restrictions. Requirements must address conflicts between desirable properties.
Training can then encourage preferred behavior. Ouyang and colleagues demonstrated a sequence involving supervised demonstrations, rankings of alternative responses and reinforcement learning from human feedback. Bai and colleagues investigated principle-guided critiques, revisions and AI-generated preference feedback in Constitutional AI. These approaches do not solve alignment.
Evaluation asks whether improvements hold on independent tasks, difficult cases and changed conditions. Operational controls, monitoring and correction remain necessary because training examples and feedback are incomplete representations of the intended behavior.

Three practical examples

Customer support: An assistant should solve legitimate problems while respecting refund authority. Agreeing to every request might please users but exceed its permitted role.
Personal planning: A scheduling assistant should respect the user's priorities and ask before making consequential changes. Filling every available hour would not necessarily serve the person's intentions.
Research support: A summarizer should represent evidence faithfully. Producing a confident, elegant answer that invents findings fails the intended purpose even if readers initially prefer its style.

Think of it this way

Imagine asking someone to make a room comfortable. Turning the heating to its maximum satisfies a crude interpretation of “make it warmer,” but may defeat the actual goal. Alignment is concerned with that gap between a measurable instruction and the wider intention. Unlike a thoughtful human helper, a model cannot be assumed to resolve the gap reliably.

Worked example

Suppose a fictional assistant is evaluated on 100 support requests. Version A closes 90 requests, but reviewers find that only 72 of those closures genuinely resolve the problem. Version B closes 85 requests, of which 80 are verified resolutions.
If the target is simply closure rate, A scores 90% and B scores 85%. If the intended outcome is verified resolution across all requests, A scores 72% and B scores 80%. B performs better on that outcome despite closing fewer requests.
Among closed requests, verified resolution is 72 / 90 = 80% for A and 80 / 85, approximately 94.1%, for B. These conditional percentages answer a different question from the overall resolution rate.
This illustrates an inadequate proxy objective. It does not prove that B is fully aligned: neither calculation measures privacy, fairness, honesty about uncertainty or consequences beyond the test. The review criteria themselves also need scrutiny.

Illustrative case study

Fictional company Cedar Guides builds an assistant for explaining product instructions. Early evaluation rewards answers that reviewers find fluent and complete. The team notices that the assistant sometimes fills gaps with plausible details absent from the approved manual.
It revises the requirements so that acknowledging missing information can be preferable to an apparently complete answer. Reviewers compare responses against the relevant manual, and the test set includes incomplete, conflicting and out-of-scope questions. The product also offers a route to a human specialist.
The team keeps helpfulness and source fidelity as separate measures and examines cases where they conflict. This fictional case illustrates how a preference for polished answers can obscure the intended task. It supplies no claim that a particular training method eliminates unsupported statements.

Common misconceptions and mistakes

“An aligned model obeys every user.” Appropriate behavior may require refusing, clarifying or escalating a request.
“High capability means good alignment.” Skill at producing outputs does not establish that those outputs serve the intended goals.
“Human feedback represents everyone's values.” Feedback reflects the people, instructions, examples and aggregation choices used to collect it.
“A passed benchmark proves alignment.” Tests cover selected behaviors under selected conditions. Broader claims require broader evidence.
“Alignment is just a prompt.” Instructions can influence behavior, but alignment also involves objectives, data, training, evaluation and how the system is used.

Business use

Alignment gives teams a useful way to question whether their optimization targets match the outcomes they actually want. Specify the intended service, affected groups, prohibited actions and escalation conditions. Compare supplier claims with evidence from the organization's own use case.
Maintain a route for reporting failures and changing requirements. A model update, new tool permission or different user population can change behavior even when the original intent remains the same. Operational approval should describe the tested scope and its limits.

Check your understanding

Why is closure rate inadequate in the worked example?

Answer: It counts unresolved closures as successes.

Does preferring a response establish that it is truthful?

Answer: No. Preference can reflect style or other qualities.

Why test after changing tool permissions?

Answer: The assistant can now produce different consequences, so earlier evidence may no longer cover its operating conditions.

Related terms

AI safety
Reinforcement Learning from Human Feedback
Constitutional AI
AI governance
reward misspecification
human oversight
model evaluation
reward hacking

Learn next

Compare alignment objectives with the feedback signals used to train a model. Then examine reward misspecification and evaluation design, paying particular attention to cases where a good score hides an unwanted outcome.

Sources

Ouyang and colleagues, Training language models to follow instructions with human feedback, original training research. https://arxiv.org/abs/2203.02155 Bai and colleagues, Constitutional AI: Harmlessness from AI Feedback, original principle-guided training research. https://arxiv.org/abs/2212.08073 NIST AI RMF 1.0, Core, lifecycle governance and evaluation context. https://airc.nist.gov/airmf-resources/airmf/5-sec-core/ The support calculations, practical examples and Cedar Guides case are original teaching illustrations.

Previous: Aliasing (factorial experiments) Next: Alignment method