Does it require perfectly optimal demonstrations?

The probabilistic formulation can represent variation and imperfect behavior, although its assumptions still need scrutiny.

Are equal-reward trajectories equally likely?

In the stated finite deterministic formulation with the same base measure, yes.

Can the inferred reward be used for planning?

Yes, but planning quality depends on the learned model and environment assumptions.

AI Glossary / Reinforcement learning and robotics / Advanced

Maximum entropy inverse reinforcement learning

Definition

Maximum entropy inverse reinforcement learning is a method for learning a reward model from demonstrated sequential behavior. It chooses a distribution over feasible trajectories that matches selected statistics of the demonstrations while making as few additional preferences as possible. In the standard deterministic formulation, trajectory probabilities are proportional to the exponential of their learned total rewards.

What it means

Ordinary reinforcement learning starts with rewards and learns behavior. Inverse reinforcement learning starts with behavior and seeks rewards that help explain it. Many reward functions can explain similar demonstrations, and many behavior distributions can match the same measured features.

The maximum entropy principle addresses ambiguity in the distribution of behavior: among distributions satisfying the selected constraints, prefer the one with greatest entropy. This does not establish the demonstrator's true motives. The result is an explanatory model built from particular features, possible trajectories and assumptions about the environment.

Entropy is maximized subject to evidence, not without constraints.

How it works

Define features of a trajectory, such as its accumulated distance, number of turns or time spent in particular conditions. Estimate their average counts from demonstrations. In the basic linear reward model, total reward is a weighted sum of these features.

For a finite deterministic trajectory space, assign each trajectory probability proportional to exp(theta dot f), where theta contains reward weights and f contains trajectory features. A partition function sums these weights over all feasible trajectories so probabilities add to one.

Fit theta by maximizing demonstration likelihood. For the basic model, the average log-likelihood gradient is empirical feature counts minus model-expected feature counts. Computing the latter can use dynamic programming rather than enumerating every trajectory. Regularization can accommodate limited evidence and prevent extreme estimates.

Specify starts, goals, horizon and termination carefully: these define which trajectories compete for probability. Stochastic dynamics require additional treatment. The original paper's approximation for that setting should not be confused with later maximum causal entropy formulations that explicitly respect the timing of decisions and random outcomes.

Three practical examples

1. A route-preference model learns how demonstrated trips trade distance against turns, then predicts probabilities for alternative feasible routes.

2. A simulated manipulation study infers a reward over movement features from demonstrations, checking predictions on tasks excluded from training.

3. An analysis of workflow sequences examines whether observed process choices can be explained by selected delay and effort features, without treating the inferred weights as proven intentions.

Think of it this way

Think of completing a picture from a few measured properties. If two routes look identical under every feature the evidence constrains, the model has no basis for arbitrarily favoring one. Maximum entropy spreads the remaining uncertainty. Adding a meaningful feature can change that conclusion; adding unsupported certainty cannot.

Worked example

Consider an original finite deterministic example with three feasible trajectories, A, B and C, sharing the same start and goal. Define one feature: whether the trajectory passes through a designated zone. Its values are 0 for A and 1 for both B and C.

In 100 demonstrations, 80 trajectories pass through the zone. The empirical feature average is therefore 0.8. Matching it requires p(B) plus p(C) to equal 0.8, leaving p(A) equal to 0.2.

Many distributions meet this constraint. One is (0.2, 0.8, 0), which arbitrarily gives C no probability. The maximum entropy distribution instead divides the indistinguishable remaining mass equally: (0.2, 0.4, 0.4).

Its entropy, using natural logarithms, is minus 0.2 ln(0.2) minus twice 0.4 ln(0.4), approximately 1.05492 nats. The concentrated alternative has entropy minus 0.2 ln(0.2) minus 0.8 ln(0.8), approximately 0.50040 nats.

The exponential reward model reproduces the maximum entropy solution with theta equal to ln(2). The unnormalized trajectory weights are then 1, 2 and 2. Their sum is 5, giving probabilities 0.2, 0.4 and 0.4. The model feature expectation is 0.8, so the unregularized feature-matching gradient is zero.

This example deliberately uses only one feature. Even if the 80 zone visits happened to split unevenly between B and C, the model cannot explain that distinction without an additional feature or assumption. Matching selected statistics is weaker than reproducing every detail of the demonstrations.

Illustrative case study

In a fictional project, Maple Routes studies simulated delivery journeys. The team initially models only journey length and number of turns. Its inferred reward predicts some routes well but repeatedly misses journeys that avoid a congested crossing.

Review reveals that congestion was absent from the features. The team adds a time-dependent congestion measure and evaluates predictions on separate journeys. It describes the result as a model of observed route choices, not a discovery of drivers' private preferences. Before using the model for recommendations, it separately checks operational objectives, because copying historical behavior is not automatically the best future decision.

Common misconceptions and mistakes

Maximum entropy does not mean maximum randomness without constraints. Demonstration statistics shape the resulting distribution.

The method does not generally recover unique true reward weights. Redundant features, omitted factors and reward equivalences can leave ambiguity even when predictions fit well.

Global trajectory normalization differs from independently applying a softmax to immediate action scores. Future branching and feasible completions matter. Also avoid treating a simple exponential path distribution as an exact solution for every stochastic environment. Finally, demonstrations may contain systematic errors or selection effects; high likelihood alone does not certify desirable behavior.

Business use

The method can help organizations model sequential choices when writing a reward function directly is difficult. Useful evaluation asks whether it predicts held-out behavior, transfers across environments and remains interpretable under plausible feature changes. Separate descriptive goals, such as understanding existing choices, from prescriptive goals, such as recommending better ones.

Check your understanding

Why do B and C receive equal probability?

Answer: Their constrained feature descriptions are identical.

What does a zero feature-matching gradient prove here?

Answer: That the chosen one-feature model matches the demonstrated average, not that it explains every observed choice.

Related terms

Inverse reinforcement learning
Imitation learning
Maximum entropy
partition function
feature matching
reward ambiguity and maximum causal entropy

Learn next

Next, add a second feature distinguishing B from C and recompute the constraints. Then study how causal entropy handles stochastic transitions without assigning the decision-maker control over random outcomes.

Sources

Primary source: Brian D. Ziebart and colleagues, Maximum Entropy Inverse Reinforcement Learning, author-hosted corrected version: https://www.cs.cmu.edu/~bziebart/publications/maxentirl-bziebart.pdf

The author publication page identifies the correction: https://www.cs.cmu.edu/~bziebart/publications/maximum-entropy-inverse-reinforcement-learning.html

The three-trajectory calculation and fictional case are original teaching material. The worked example is finite and deterministic, with a single explicitly defined feature.

Previous: Maximum entropy Next: Maximum-entropy Markov model