Can the KL exceed its target?

Yes; the target guides adaptation rather than imposing a hard cap.

Why increase beta after an oversized update?

To make comparable divergence more costly in the next optimization phase.

Is the original factor-of-two rule mandatory?

No, but a different rule should be documented rather than silently treated as identical.

AI Glossary / Reinforcement learning and robotics / Advanced

PPO-Penalty

Definition

PPO-Penalty is a proximal policy optimization variant that subtracts a penalty for KL divergence between the old and candidate policies from a policy-improvement surrogate. Its adaptive form increases or decreases the penalty coefficient according to observed policy change. It seeks useful updates while discouraging excessive drift from the policy that collected the training batch.

What it means

Improving a sampled policy objective aggressively can move the learner into behavior poorly represented by its data. PPO-Penalty places a cost on that movement. A candidate update must offer enough apparent benefit to justify the statistical distance it creates from the old policy.

The penalty coefficient controls this tradeoff. A larger coefficient makes the same policy divergence more expensive; a smaller one permits greater movement for the same surrogate gain.

A penalty is not a hard constraint. A candidate can exceed the desired divergence if the optimizer or objective favors doing so, and adapting the coefficient usually affects a later update rather than undoing the one already taken.

How it works

Collect a batch with an old policy and estimate its advantages. For each recorded action, form the ratio of its current policy probability to its old probability. The unpenalized surrogate averages that ratio multiplied by the fixed advantage estimate.

Subtract beta times the average KL divergence from the old action distribution to the new action distribution at sampled states. The KL direction matters: reversing its arguments generally changes the numerical penalty and optimization behavior.

Optimize this objective for several minibatch epochs, keeping the old-policy reference fixed. Then measure the resulting average divergence d. In the original paper's simple adaptive rule, halve beta if d is below the target divided by 1.5, and double beta if d exceeds the target multiplied by 1.5. Otherwise keep beta unchanged. The updated coefficient is used for the next policy update.

This is one documented adaptation rule, not a universal definition of every implementation. Value losses, entropy terms, advantage estimation and optimization settings must also be specified. The measured KL is averaged over sampled states, so even a small batch average does not bound every state's change.

Three practical examples

1. A simulated control experiment adjusts its penalty coefficient when measured policy changes repeatedly exceed the intended scale.

2. A researcher compares adaptive KL penalties with ratio clipping while keeping rollout collection and advantage estimation comparable.

3. An implementation review checks that the KL uses full action distributions or a clearly specified estimator rather than confusing one observed action's log ratio with an exact distributional divergence.

Think of it this way

Think of adding a movement charge to an improvement proposal. A larger charge can make a modest revision more attractive than a dramatic revision with greater apparent benefit. Adjusting the charge after reviewing results changes future incentives; it does not make the previous revision physically impossible or guarantee that its consequences were beneficial.

Worked example

Consider an original one-state, two-action example. The old policy is (0.5, 0.5), and the fixed advantages are +2 for A and -2 for B. Use the exact old-policy average to make the calculation transparent.

For candidate policy (0.6, 0.4), the ratios are 1.2 and 0.8. The surrogate is 0.5 times 1.2 times 2 plus 0.5 times 0.8 times -2, giving 0.4. The KL divergence from old to new is 0.5 ln(0.5/0.6) plus 0.5 ln(0.5/0.4), approximately 0.020411 nats.

With beta equal to 5, the penalized objective is 0.4 minus 5 times 0.020411, approximately 0.297945.

Now consider the more aggressive candidate (0.8, 0.2). Its surrogate is 1.2, but its KL divergence is approximately 0.223144 nats. With beta 5, its penalized objective is approximately 0.084282. Among these two candidates, the penalized objective prefers the moderate change even though its unpenalized surrogate is smaller.

Set the target KL to 0.01. The moderate candidate's measured divergence, 0.020411, exceeds 1.5 times that target, or 0.015. Under the paper's adaptation rule, beta would double from 5 to 10 for the next update.

For comparison, a candidate (0.52, 0.48) has KL approximately 0.000801, below 0.01 divided by 1.5. Starting from beta 5, that outcome would instead halve beta to 2.5. These are separate illustrative adaptation cases, not consecutive updates of one run. None of the objective comparisons establishes actual reward improvement outside the stipulated advantage model.

Illustrative case study

In a fictional project, Alder Control observes that its early policy updates barely change action probabilities, while later updates become much larger. The team introduces an adaptive KL penalty and logs both beta and achieved divergence.

The logs reveal occasional overshoots despite the target. The report explains that coefficient adaptation is a feedback mechanism, not an exact guarantee. Evaluation episodes determine whether the resulting policy performs better, and a separate diagnostic examines states with unusually large changes hidden by the average.

Common misconceptions and mistakes

Do not confuse beta with the KL target. Beta determines the cost of divergence; the target guides its adaptation.

PPO-Penalty does not directly clip ratios or enforce a strict trust-region boundary. It also does not imply that each state's KL equals the target after optimization.

Keep the KL direction and old-policy reference explicit. Changing either alters the objective. A low penalty term may reflect small measured change on the sampled states while saying little about poorly covered states. Finally, numerical superiority of a surrogate is not direct evidence of higher realized reward.

Business use

The method is useful for studying how controlled policy changes affect learning stability and use of recent experience. Business-facing evaluation should report task performance, variation across runs and the interaction cost of collecting data. The adaptive coefficient is a training control, so reviewers should ask how it behaves over time and whether repeated overshoots indicate unsuitable optimization settings or unreliable advantages.

Check your understanding

Why does the moderate candidate win when beta is 5?

Answer: Its much smaller divergence more than offsets its smaller surrogate benefit.

Which KL direction was used?

Answer: Old policy to candidate policy.

Related terms

Proximal policy optimization
PPO-Clip
Trust region
KL divergence
surrogate objective
advantage estimation and policy gradient

Learn next

Next, repeat the candidate comparison with beta equal to 1. Then reverse the KL arguments and inspect why the two divergence values differ.

Sources

Primary source: John Schulman and colleagues, Proximal Policy Optimization Algorithms, Section 4 on the adaptive KL penalty: https://arxiv.org/pdf/1707.06347

The categorical-policy calculations and fictional case are original teaching material. The adaptation examples use the specific rule presented in the paper and do not claim exact trust-region enforcement.

Previous: PPO-Clip Next: Practical significance