Does MPO require differentiating Q with respect to an action?

Its nonparametric reweighting can compare sampled action values without that derivative.

Is q the deployed policy?

Usually the parameterized policy fitted in the M-step is deployed.

Does off-policy mean arbitrary historical data are sufficient?

No. Coverage and reliable policy evaluation remain essential.

AI Glossary / Reinforcement learning and robotics / Advanced

Maximum a posteriori policy optimization

Definition

Maximum a posteriori policy optimization, usually called MPO, is a reinforcement-learning method that improves a policy through alternating action reweighting and policy fitting. It uses estimated action values to favor promising choices, then trains a parameterized policy to reflect those preferences while limiting how far the policy changes. The original method supports learning from off-policy experience.

What it means

A policy is a rule for choosing actions. MPO separates improving that rule into two manageable questions: which available actions look better according to the current value estimate, and how should the policy model learn those preferences?

The first answer is an auxiliary distribution over actions. The second is a fitted policy that can generalize across states. These objects need not be identical. A neural policy may be unable to represent the auxiliary distribution exactly, and constraints can deliberately restrict the fit.

The maximum a posteriori name comes from the probabilistic-inference derivation and regularized policy fitting. It does not mean the algorithm discovers a uniquely correct policy or supplies calibrated uncertainty about every operational outcome.

How it works

First, learn an action-value function for the current policy using suitable policy-evaluation machinery and stored experience. The original implementation uses a Retrace-based critic. Errors in this critic affect the preferences used in the next stage.

In the nonparametric E-step, improve an auxiliary action distribution while bounding its average KL divergence from the current policy. At a given state, the ideal reweighting has q proportional to the old policy probability multiplied by exp(Q divided by eta). The positive temperature eta is obtained through a dual optimization associated with the constraint.

In the M-step, fit the parameterized policy to these weighted actions. An additional constraint limits change from the previous policy. The original E-step and M-step use different KL directions, so their budgets and divergences must not be casually interchanged.

Repeat evaluation, reweighting and fitting. Finite samples, approximate critics and imperfect optimization distinguish practical training from idealized improvement arguments. A trust region constrains a distributional change, not every possible consequence of deploying the policy.

Three practical examples

1. A simulated robot-control project compares candidate joint commands using an action-value critic and fits a stochastic controller to the preferred commands.

2. A research team reuses stored transitions to evaluate successive policies while tracking whether replay data still cover relevant situations.

3. A training diagnostic examines whether an overly concentrated action distribution comes from exaggerated critic values or an overly permissive update budget.

Think of it this way

Think of a coach who first ranks practice options, then gives a learner a manageable revised routine. The ranking can be more ambitious than the routine the learner adopts immediately. Both stages rely on the coach's assessments; carefully limiting a change does not make an incorrect assessment reliable.

Worked example

Consider an original one-state, two-action illustration of the reweighting step. The current policy selects A with probability 0.75 and B with probability 0.25. Let the critic values be Q(A) equal to 0 and Q(B) equal to ln(3), approximately 1.0986. For this illustration, take eta equal to 1.

The unnormalized weights are 0.75 times exp(0), which is 0.75, and 0.25 times exp(ln(3)), also 0.75. Normalizing gives q(A) equal to q(B) equal to 0.5.

The KL divergence from q to the old policy is 0.5 ln(0.5/0.75) plus 0.5 ln(0.5/0.25), approximately 0.14384 nats. Thus this auxiliary distribution fits an E-step budget of 0.15 nats. We have chosen eta to illustrate the calculation, not solved the dual for that particular budget.

The critic-based expected value rises from 0.25 ln(3), approximately 0.27465, to 0.5 ln(3), approximately 0.54931. This is improvement under the stipulated critic, not a measured increase in real reward.

Suppose a candidate fitted policy is (0.6, 0.4). Its KL divergence in the M-step direction, from the old policy to the candidate, is 0.75 ln(0.75/0.6) plus 0.25 ln(0.25/0.4), approximately 0.04986 nats. It fits a separate 0.05-nat budget. This candidate demonstrates a feasible partial fit; the calculation does not claim it is the exact constrained optimizer or a complete MPO training iteration.

Illustrative case study

In a fictional project, Linden Robotics trains a controller entirely in simulation before staged testing. Its first experiments show that the reweighted action distribution strongly prefers commands rarely present in replay data.

The team checks whether the critic is extrapolating, examines separate evaluation episodes and tightens its experiment controls. It records the auxiliary preferences and the fitted policy separately, allowing reviewers to see how much of each proposed change was adopted. The project reports actual task outcomes alongside training objectives, because a favorable weighted fit can coexist with poor performance when value estimates are wrong.

Common misconceptions and mistakes

Do not equate MPO with ordinary supervised imitation of recorded actions. Its fitting targets are reweighted using learned values.

Do not confuse MPO with PPO or its distinct on-policy relative V-MPO. Names and shared trust-region ideas do not make their objectives interchangeable.

An average KL constraint does not cap every state's divergence. It also does not guarantee safe actions, reliable rewards or freedom from exploitation of a flawed simulator. Theoretical improvement results depend on assumptions that a practical neural implementation may only approximate.

Business use

MPO is relevant to research and development involving sequential decisions, particularly when reusing experience can reduce costly interactions. Evaluation should include environment steps, computation, variation across runs and performance under changed conditions. Decision-makers should ask how critic reliability and replay coverage are assessed, and whether operational limits are enforced separately from the statistical update constraints.

Check your understanding

Why did B's probability increase despite its smaller original probability?

Answer: Its exponential value multiplier compensated for that difference.

Why are there two KL calculations?

Answer: They describe distinct constraints with distinct directions and roles.

Related terms

Policy iteration
Expectation-maximization
Trust region
Experience replay
action-value function
KL divergence
Retrace and V-MPO

Learn next

Next, repeat the two-action calculation with a larger temperature. Then examine how weighted maximum-likelihood fitting projects an auxiliary distribution into a restricted policy family.

Sources

Primary source: Abbas Abdolmaleki and colleagues, Maximum a Posteriori Policy Optimisation, particularly Sections 3 and 4: https://arxiv.org/pdf/1806.06920

The numerical example and fictional case are original teaching material. The example isolates reweighting and feasibility checks; it is not a full dual optimization or an experimental performance claim.

Previous: Maximum a posteriori Next: Maximum entropy