Does the robot know the reward at the start?

In the defining setup, the human knows the relevant reward parameter while the robot has uncertainty about it.

Must learning be passive observation?

No. Interaction can include informative human actions and agent actions that help reveal the objective.

Does CIRL solve AI alignment completely?

No. It formalizes a particular cooperative problem under explicit assumptions; broader alignment and deployment questions remain.

AI Glossary / Causal inference and experimentation / Advanced

Cooperative inverse reinforcement learning

Definition

Cooperative inverse reinforcement learning (CIRL) is a framework in which a human and an artificial agent share an objective, but the agent is uncertain about the reward function specifying that objective. Their interaction can include actions that teach the agent what the human values.

What it means

Ordinary demonstrations can be interpreted as attempts to perform a task well. Teaching demonstrations have another purpose: helping a learner understand the task. Someone may take a slower route through an example because that route reveals an important distinction.

CIRL makes this interaction part of the decision problem. The artificial agent is not merely watching a fixed dataset; the human and agent can choose actions whose informational consequences affect their later cooperation.

The original framework assumes a human who knows the relevant reward parameter and an agent that does not. This is a mathematical simplification. Real people can be uncertain, inconsistent or in disagreement, and the framework's shared-reward assumption does not resolve those difficulties by itself.

How it works

A CIRL model specifies world states, available actions, transitions, possible reward parameters and a prior belief about those parameters. Both participants receive the same reward, while their information differs.

The agent updates its beliefs from the interaction. When planning, an action can matter both because it changes the world and because it reveals something about the objective. The human's policy can likewise take account of what the agent is likely to learn.

Hadfield-Menell and colleagues formalized CIRL in 2016 and showed a reduction of optimal joint-policy computation to a partially observable Markov decision process. This is a structural result under the model's assumptions, not a guarantee that large real-world cooperation problems are easy to solve. Coordination and inaccurate models of human behavior remain important practical issues.

Three practical examples

A research assistant asks which of two trade-offs matters before choosing a longer sequence of work steps.
A robot-training demonstration deliberately includes an unusual object so the learner can distinguish a general preference from a habit.
A scheduling prototype models how a user's correction can reveal priorities that affect several future choices.

Think of it this way

Think of an apprentice watching a craftsperson who knows they are teaching. The craftsperson may pause at a subtle step, while the apprentice asks a question before continuing.

The analogy helps explain why instruction changes behavior. It does not imply that CIRL agents possess human understanding or that every instructional conversation implements this particular formal model. Those claims would require separate evidence.

Worked example

When clarification has value

Consider an original two-option teaching scenario. A person has one of two known preference types: quiet-first or speed-first, each initially assigned probability 0.5 by an assistant. The person knows their type.

Action Q gives reward 10 to a quiet-first person and 0 to a speed-first person. Action S reverses those rewards. Both participants are assumed to share that reward. Acting immediately yields expected reward 0.5 × 10 + 0.5 × 0 = 5 for either choice.

Suppose the assistant can first ask a perfectly informative question, and the cost of asking is 1 reward unit. A truthful answer allows the matching action, giving net reward 10 − 1 = 9. Under these assumptions, asking is preferable.

If the cost becomes 6, the net is 4 and immediate action is better among these options. If answers are noisy, their expected value must also be reconsidered. This simplified value-of-information calculation illustrates one cooperative interaction; it is not a reproduction of the original paper's algorithm or a complete human-preference model.

Illustrative case study

Fictional case: an office assistant prototype

A fictional team builds an assistant that prepares meeting arrangements. Early tests show that a user's repeated choice of a nearby room can have several explanations: short walking distance, quiet surroundings or particular equipment.

The team designs a controlled study in which participants can demonstrate or explain their priorities. It compares a passive learner with a prototype that can request clarification before proposing arrangements. Its evaluation includes whether clarification actually changes useful decisions and whether participants understand the question.

The prototype is described as inspired by cooperative preference learning. The team would need to specify and test a CIRL model before claiming that the product implements CIRL. No business improvement or deployment outcome is assumed in this fictional case.

Common misconceptions and mistakes

CIRL does not mean that the agent should copy every human action. A demonstration can contain teaching signals, mistakes or constraints that differ from the agent's own situation.

It is also not synonymous with reinforcement learning from human feedback. The approaches can share an interest in human objectives, but CIRL specifically models cooperative interaction with asymmetric information about a shared reward.

Finally, a single human's reward function is not a complete social mandate. Conflicting stakeholders, changing preferences and actions outside the model require additional design and governance choices. A mathematical optimum cannot settle which people's interests should be represented.

Business use

CIRL is useful as a design lens when an assistant must learn preferences while working with a person. Ask whether the interface lets people teach, correct and clarify, and whether the system models those signals appropriately.

For a practical pilot, document the possible preference hypotheses, what observations distinguish them, the cost of asking and the consequences of a mistaken inference. Test cases in which the human behaves differently from the assumed teaching policy. Retain an understandable account of why a clarification was needed and how it affected the proposed action.

Check your understanding

In the worked scenario, perfect clarification costs 3 reward units. Should the assistant ask, compared with immediate expected reward 5?

Answer: Yes, under the stated assumptions. Clarification produces 10 − 3 = 7, exceeding 5. This conclusion changes if information quality, available actions or reward assumptions change.

Related terms

Learn next

Compare passive inverse reinforcement learning with an instructional demonstration. Then study how a POMDP represents uncertainty and why a model of the teacher affects what a learner infers.

Inverse reinforcement learning

Sources

Primary source: Hadfield-Menell, Dragan, Abbeel and Russell, Cooperative Inverse Reinforcement Learning, NeurIPS 2016: https://papers.neurips.cc/paper_files/paper/2016/file/c3395dd46c34fa7fd8d729d8cf88b7a8-Paper.pdf

The clarification calculation and office-assistant case are original explanatory examples.

Previous: Cooperative coevolution Next: Cooperative storage cloud