Does AIRL only copy actions?

Its aim includes learning a reward representation, although imitation is part of training.

Can the reward transfer to new dynamics?

Sometimes, under suitable conditions; transfer needs evaluation.

Is the discriminator output the extracted reward?

No. The formulation distinguishes the discriminator, shaped score and reward component.

AI Glossary / Reinforcement learning and robotics / Advanced

Adversarial inverse reinforcement learning

Definition

Adversarial inverse reinforcement learning, commonly abbreviated AIRL, is a method for learning a reward function from expert demonstrations using an adversarial training process. It alternates between distinguishing demonstrated behavior from a learner's behavior and improving the learner. Its structured reward model aims to separate the task reward from additional terms that help explain behavior in the training environment.

What it means

Ordinary reinforcement learning starts with a reward specification and seeks behavior that performs well under it. Inverse reinforcement learning works in the other direction: it observes demonstrations and tries to infer a reward that could explain them. This inference is ambiguous because different rewards can support the same behavior.
AIRL addresses part of that ambiguity through the structure of its discriminator. The intended benefit is a learned reward that can sometimes support learning in changed environments, rather than merely reproducing the original action choices. This is a research objective with assumptions and experimental limits, not a promise that demonstrations reveal a person's unique intentions.

How it works

The learner collects trajectories under its current policy. A discriminator is trained to distinguish those samples from expert demonstrations. The policy is then improved using a signal derived from the discriminator, and the process repeats.
In the structured AIRL formulation, a score f combines a reward component g with a potential-based term: f(s, a, s_next) = g(s, a) + gamma h(s_next) - h(s). The discriminator uses D = exp(f) / (exp(f) + pi(a|s)). For continuous actions, the policy quantity is a density; it is not a probability mass for an exact action.
The policy-training signal in the paper's algorithm is log D - log(1-D), which equals f - log pi(a|s). This signal, the score f and the extracted reward component g are distinct quantities. Restricting g to depend only on the current state supports the paper's reward-recovery analysis under its stated conditions. General training with arbitrary demonstrations, dynamics and function approximators does not automatically satisfy those conditions.

Three practical examples

Simulated movement: researchers infer a reward from demonstrations and then train a new policy after changing aspects of the simulated dynamics. They evaluate transfer rather than assuming imitation quality proves reward recovery.
Demonstration analysis: a team compares behavior learned under an inferred reward with the demonstrations that motivated it, checking where the model fails to explain observed choices.
Reward-model comparison: researchers compare a structured AIRL model with less constrained alternatives while holding demonstrations and evaluation tasks fixed. Different reward representations can fit behavior similarly yet behave differently after an environment change.

Think of it this way

Imagine observing someone repeatedly taking a particular route. They might prefer the destination, avoid a difficult slope or simply know that route best. Copying their turns does not establish which preference matters. AIRL attempts to learn a useful explanation in reward form, but observations alone can leave several explanations possible.

Worked example

Consider one illustrative discriminator evaluation, without training. Use a discrete action with policy probability 0.25. Suppose the reward component is g = 0.2, the current potential is h(s) = 0.4, the next potential is h(s_next) = 0.5 and gamma = 0.9.
The shaping contribution is 0.9 times 0.5 - 0.4 = 0.05. The combined score is therefore f = 0.2 + 0.05 = 0.25. Its exponential is approximately 1.28403. Substituting into the discriminator gives D = 1.28403 / (1.28403 + 0.25), approximately 0.83703.
The policy-training signal is log(0.83703) - log(1-0.83703), approximately 1.63629, using natural logarithms. The algebraic shortcut confirms this: f - log pi = 0.25 - log(0.25) = 0.25 + 1.38629 = 1.63629.
These three numbers have different roles. The reward component is 0.2, the shaped score is 0.25 and the discriminator log-odds signal is approximately 1.63629. Calling all three the recovered reward would obscure how the method works.
Now keep g, both potentials and gamma fixed but change the policy probability to 0.5. The combined score remains 0.25. The discriminator becomes 1.28403 / (1.28403 + 0.5), approximately 0.71973, and its log-odds signal becomes 0.25 - log(0.5), approximately 0.94315. Thus changing the policy changes the discriminator output and training signal even with the same reward component.
Nothing in this arithmetic demonstrates that g is the correct task reward. It only checks the relationships among quantities in the specified formulation. Demonstrations, training, evaluation and the assumptions needed for a recovery claim are absent from this teaching example.

Illustrative case study

Fictional company Alder Simulation studies a demonstration-based controller in a virtual environment. It trains a structured reward model and separately evaluates a policy optimized under the extracted reward after changing the simulator's dynamics. It compares task outcomes rather than relying solely on discriminator accuracy.
When transfer performance falls, the analysts inspect demonstration coverage, model assumptions and the reward representation. They do not conclude that a convincing imitation established the demonstrator's true objective. This fictional workflow reports no measured results.

Common misconceptions and mistakes

Adversarial training does not mean that the system is attacking a user. Here it describes the competition between a discriminator and a policy during learning. A discriminator that separates samples well is also not, by itself, proof of a useful recovered reward.
AIRL should not be presented as recovering a unique reward in every setting. The paper's stronger recovery results involve restrictions, including state-only rewards and conditions on dynamics; finite-data neural training adds further uncertainty. Another mistake is transferring the entire policy-training signal as though it were the extracted reward component. Finally, demonstrations can contain inconsistent or suboptimal behavior that the assumed expert model does not describe well.

Business use

AIRL can inform research where demonstrations are easier to obtain than a satisfactory reward specification. Practical evaluation should include the cost of environment interaction, demonstration quality and performance under intended changes. An inferred reward remains a model to test, not an authoritative statement of customer preferences or organizational goals.

Check your understanding

Why did D change in the example?

Answer: The policy probability in its denominator changed.

Why was g still 0.2?

Answer: Its stipulated value was held fixed.

Why is imitation insufficient proof of reward recovery?

Answer: Different rewards can explain similar behavior.

Related terms

Inverse reinforcement learning
Imitation learning
Reward shaping
Discriminator
Expert demonstration
Policy optimization
Reward ambiguity

Learn next

Compare the extracted reward with the training signal, then design a transfer evaluation that changes dynamics while retaining the intended task.

Sources

Fu, Luo and Levine, Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, ICLR 2018, Algorithm 1 and Sections 5–6: https://arxiv.org/pdf/1710.11248 . The numerical example and company scenario are original illustrations.

Previous: Adversarial example Next: Adversarial loss