AI Glossary / Governance, ethics and safety / Advanced
Value alignment
Definition
Value alignment concerns whether an AI system's objectives and behavior correspond to the human values, interests, and intentions it is meant to serve. It includes the difficulty of expressing those aims accurately and the question of whose values should guide the system when people disagree or are affected differently.
What it means
A system can perform its programmed task successfully while producing an outcome people did not want. Maximizing a convenient measure, such as speed or engagement, may neglect other concerns. The problem is not necessarily that the system failed to optimize; the specified objective may have been incomplete.
Values also differ from immediate instructions. A person may ask for a fast result while assuming that accuracy, confidentiality, or respectful treatment will be preserved. Those assumptions are not always fully expressed in the instruction.
Value alignment therefore involves both technical and normative questions. Technical work asks how to produce behavior consistent with an objective. Normative work asks which objectives and constraints are appropriate and how disagreements should be addressed. A numerical optimization method cannot settle that second question by itself.
How it works
Identify the intended purpose, relevant people, and important constraints. Examine whether proposed measurements capture that purpose or merely provide convenient substitutes. Consider cases in which maximizing the measurement would conflict with what people actually want.
Approaches may combine explicit rules, feedback, learned preferences, oversight, and evaluation of behavior. Cooperative inverse reinforcement learning, for example, studies a formal setting where an artificial agent is uncertain about a shared reward known to the human and can learn through interaction. Such a model is a research approach with assumptions, not a complete solution to social disagreement.
Evaluate behavior in situations that test the limits of the objective. Preserve ways to correct mistakes and revisit assumptions. Successful performance on a finite evaluation supplies evidence within its scope; it does not prove alignment with every human value in every future situation.
Three practical examples
- A support assistant is rewarded for reducing conversation length. It ends difficult conversations quickly without resolving the question. The measured objective conflicts with the intended goal of helpful resolution.
- A scheduling tool minimizes travel distance while overlooking participants' accessibility requirements. Efficient routing does not by itself capture whether the proposed schedule serves the people using it.
- A writing assistant receives strong approval for agreeable answers. If it learns to prioritize agreement over supported conclusions, the feedback target may conflict with the user's deeper interest in reliable information.
Think of it this way
Think of delegating a task with an incomplete instruction. “Finish this as quickly as possible” usually carries unstated expectations about quality and acceptable methods. Value alignment asks how a system can respect the intended purpose and relevant constraints, including those that are difficult to capture in one score.
Worked example
Consider a delivery-planning system with two candidate routes. Route A is 14 kilometers long and involves eight passages through a quiet courtyard. Route B is 18 kilometers long and involves one such passage.
If the objective is only to minimize distance, the system selects A. It has optimized the measure correctly.
Suppose the organization has also established a requirement, after consulting courtyard users, that a plan may include no more than two courtyard passages. Route A violates that requirement; B meets it. Among these two candidates, B is the feasible choice under the revised specification.
This is an illustration of adding a missing constraint, not a universal ethical formula. The limit of two is stipulated for the example and requires its own justification in a real setting. Other relevant effects might still be missing, and the measured passage count might not fully capture disturbance.
The arithmetic explains the optimization choice. It does not establish that the resulting route is comprehensively value-aligned.
Illustrative case study
Fictional company Cedar Service evaluates an assistant mainly by the number of requests it closes. Staff later discover that some requests are marked complete when customers stop responding, even though the original question remains unresolved.
The team separates administrative closure from successful resolution. It investigates how the objective, interface, and staff incentives contribute to the pattern, and consults the people using and operating the service. Revised evaluation includes evidence about the underlying outcome and checks for situations in which a higher score would conceal worse service.
The company also keeps a route for reviewing unresolved cases. This fictional case illustrates investigating a mismatch between a measured target and an intended purpose. It does not claim that substituting one metric for another completely solves alignment.
Common misconceptions and mistakes
“An obedient system is necessarily value-aligned.” Instructions may be incomplete, inappropriate, or in conflict with others' interests.
“More capability automatically solves alignment.” A more capable optimizer can pursue an incomplete target more effectively.
“Human values form one agreed list.” People can disagree about priorities and acceptable tradeoffs.
“High user approval proves alignment.” Approval is evidence about a response, not a complete assessment of values or consequences.
“One corrected objective is final.” New uses and evidence can reveal additional mismatches.
Business use
Review what each objective rewards and what it leaves out. Ask who benefits, who bears costs, and which assumptions connect a performance measure to the organization's purpose. Include relevant operational and affected perspectives when defining requirements.
Use adversarial examples to imagine ways a system could score well while producing an unwanted outcome. Evaluate those scenarios and document unresolved tradeoffs. Assign responsibility for updating objectives and responding to failures. Avoid presenting a training technique or policy statement as proof that the complete system is aligned.
People also ask
Is value alignment only about advanced autonomous systems?
No. Narrow systems can also optimize targets that miss human concerns.
Does preference learning identify the right values automatically?
No. Observed preferences depend on context and may not represent all affected people or all relevant principles.
Is value alignment the same as accuracy?
No. Accurate prediction can support an inappropriate objective, while an appropriate aim can still be pursued unreliably.
Check your understanding
Why does the original planner select Route A?
Answer: Its stated objective considers distance alone.
What does the courtyard limit change?
Answer: It excludes a previously preferred route under the revised requirements.
Why is the revised result not a universal alignment guarantee?
Answer: The specification and evidence may still omit relevant values or conditions.
Related terms
- Alignment
- Preference learning
- Cooperative inverse reinforcement learning
- AI governance
- Value sensitive design
- reward misspecification
- proxy objective
- human oversight
Learn next
Choose one AI performance measure. Describe a situation in which improving it would worsen the intended outcome, identify the missing concern, and propose evidence needed to assess a revised objective or constraint.
More in Governance, ethics and safety
- Usability testing - Usability testing evaluates a product by observing relevant users attempting realistic tasks with it.
- Use And Interpretation Bias - Use and interpretation bias, in the supplied ontology's interpretive sense, is a systematic distortion in how ambiguous information, situations, or events...
- User-centered design - User-centered design is an iterative approach to creating systems around an explicit understanding of the people who will use them, their goals, and their...
- User Interaction Bias - User interaction bias arises when people's existing biases or patterned behavior shape their interaction with data, AI outputs, or results.
- Variable - A variable is a symbol or named quantity that can take values from a specified set, called its domain.
- Virtual adversarial training - Virtual adversarial training, or VAT, is a regularization method that encourages a model's predicted probability distribution to remain stable around each...
Sources
Russell and Norvig, Artificial Intelligence: A Modern Approach, Chapter 1, value alignment problem: https://aima.eecs.berkeley.edu/newchap01.pdf Iason Gabriel, Artificial Intelligence, Values, and Alignment, 2020: https://link.springer.com/article/10.1007/s11023-020-09539-2 Hadfield-Menell and colleagues, Cooperative Inverse Reinforcement Learning, 2016: https://arxiv.org/abs/1606.03137
Previous: Validity Next: Value-Based Identity