Does PPO support continuous actions?

Yes, with an appropriate policy distribution.

Are PPO-Clip and PPO-Penalty identical?

No. The penalty variant uses a divergence penalty rather than the clipped objective described here.

Does clipping guarantee monotonic improvement?

No. Approximation, finite data, and optimization effects can still produce worse policies.

AI Glossary / Reinforcement learning and robotics / Advanced

Proximal policy optimization

Definition

Proximal policy optimization, or PPO, is a family of policy-gradient methods designed to improve a decision policy while discouraging excessively large updates from the policy that collected the training data. Its widely used clipped variant modifies the training objective using a probability ratio. This provides a practical control on optimization incentives, not a hard guarantee that the new policy stays within a fixed distance or always improves.

What it means

Changing an agent's decisions too aggressively can make useful previous experience less representative of the new behavior. PPO tries to make repeated updates from a recent batch more manageable. It generally alternates collecting experience with optimizing the policy and its value estimates.
The name does not imply that the agent has solved its task optimally. PPO remains sensitive to reward design, learning settings, and environment conditions. The clipped and penalty variants also differ, so an explanation should identify which version is being discussed.

How it works

In PPO-Clip, first collect trajectories using a fixed old policy and estimate how favorable the sampled actions were, commonly through advantage estimates. For each sampled state-action pair, form r = π_new(a|s)/π_old(a|s), using probability densities when actions are continuous.
The clipped surrogate contribution is min(rA, clip(r, 1 − ε, 1 + ε)A), where A is the fixed advantage estimate for that sample and ε controls the clipping interval. Training maximizes the average contribution, or minimizes its negative. The old-policy denominator stays fixed during the update phase.
For positive advantages, increasing action probability beyond the upper clipping threshold gives no additional benefit through that sample's clipped term. For negative advantages, sufficiently reducing action probability similarly stops giving additional benefit. Changes in the unfavorable direction are not protected by a symmetric cap on the penalty.
Implementations typically fit a value function and may include entropy terms, advantage normalization, or early stopping based on policy divergence. These details affect behavior. PPO is conventionally on-policy despite reusing a recent batch for multiple optimization passes; it is not permission to train indiscriminately on arbitrary old experience.

Three practical examples

Robot simulation: improve a stochastic controller using recent trajectories and estimated task advantages.
Game learning: update action probabilities across repeated episodes while monitoring policy changes.
Sequential allocation experiments: investigate a policy that distributes resources under a modeled reward objective and defined constraints.

Think of it this way

Think of revising a plan using a recent set of trial results, while limiting how much extra credit a single encouraging result gives to a large change. The credit limit discourages overreaction, but it is not a physical barrier that prevents the plan from changing substantially through other influences.

Worked example

Consider one fictional sampled action whose probability under the old policy is 0.20. Let ε = 0.20, so the ratio's clipping interval is [0.80, 1.20]. Treat its advantage estimate as fixed during the calculation.
First suppose A = +2 and the candidate new probability is 0.30. The ratio is 0.30/0.20 = 1.50. The ordinary surrogate is 1.50 × 2 = 3, while the clipped product is 1.20 × 2 = 2.4. Taking the minimum gives 2.4. Increasing this action's probability further gives no additional objective benefit from this sample once that ceiling is active.
If the new probability were instead 0.10, r = 0.50. The two products would be 1 and 1.6, so the contribution is 1. The unfavorable reduction in probability is not hidden by clipping it upward.
Now consider a separate hypothetical sample case with the same probabilities but A = −2. At new probability 0.10, the products are −1 and −1.6, and the minimum is −1.6. Further reducing the probability below that threshold gives no additional benefit from this term. At new probability 0.30, the products are −3 and −2.4, and the minimum is −3, preserving the penalty for making a disadvantageous action more likely.
These are individual surrogate contributions, not expected task returns or a complete training loss. Crucially, a probability ratio of 1.50 remains possible in the calculation: PPO-Clip does not forcibly set the policy's ratio to 1.20. It clips one part of an optimization objective.

Illustrative case study

Fictional case: CedarAgents trains a controller in simulation and observes occasional performance drops after an update. The team initially assumes that a clipping parameter prevents all large policy changes. It checks actual divergence and finds that the assumption is false.
The team examines learning rate, update count, value estimates, and representative evaluation runs before selecting a revised configuration. This fictional example illustrates a diagnostic process and the limits of objective clipping. It does not claim that a particular parameter setting guarantees stable learning or a better deployed controller.

Common misconceptions and mistakes

PPO clipping is not gradient clipping, reward clipping, or a strict trust-region constraint. The clipping parameter does not directly specify a maximum absolute change in every action probability. A lower policy-training loss is not itself proof of higher environment return. Multiple passes over one batch also do not make that batch fresh evidence after the policy changes substantially.

Business use

PPO can be a useful candidate for learning sequential behavior when a suitable interaction environment is available. Compare it with alternatives under consistent interaction and computation budgets. Track actual task outcomes, variability across runs, policy divergence, and value-estimation quality. Specify how sampled training actions differ from any deterministic deployment action rule, and evaluate the behavior intended for use.

Check your understanding

What ratio results when the probability changes from 0.20 to 0.30?

Answer: 1.50.

For positive advantage 2, what is the clipped contribution in that case?

Answer: 2.4.

Why is −3 retained for negative advantage at ratio 1.50?

Answer: The minimum preserves the worse contribution when an unfavorable action becomes more likely.

Related terms

Policy gradient
Clipping
Trust region
advantage
actor-critic
importance ratio
KL divergence
on-policy learning
entropy bonus
policy optimization

Learn next

Next, graph the clipped contribution separately for positive and negative advantages. Then examine why monitoring actual policy divergence can remain useful even when the clipped objective is implemented correctly.

Sources

John Schulman and colleagues, Proximal Policy Optimization Algorithms, original paper: https://arxiv.org/pdf/1707.06347 OpenAI Spinning Up, Proximal Policy Optimization, equations and clipping limitations: https://spinningup.openai.com/en/latest/algorithms/ppo.html

Previous: Proximal operator Next: Proximity search (text)