Does shaping always preserve the best policy?

No; arbitrary added rewards can change it.

Must a potential perfectly estimate future value?

No, although its usefulness for learning depends on the problem.

Can shaped reward be negative?

Yes. Negative adjustments may be necessary for the intended cancellation.

AI Glossary / Reinforcement learning and robotics / Advanced

Reward shaping

Definition

Reward shaping adds supplementary reward information to a reinforcement learning task to guide learning. It can make useful progress easier to recognize when the original task reward is sparse. However, arbitrary bonuses can change what behavior is optimal. Potential-based reward shaping is a principled construction that preserves policy preferences under the appropriate assumptions and boundary conditions.

What it means

A learner may receive no useful reward until it completes a difficult sequence. Shaping can provide intermediate feedback, such as a signal derived from a measure of progress. The purpose is to help learning while retaining the intended task objective.
The central question is whether the added signal changes the task. Rewarding every visit to a promising location, for example, can make repeated visits attractive even when they do not complete the job. A shaping score should therefore be assessed as part of the mathematical objective, not merely as helpful advice.

How it works

A potential function assigns a number Phi(s) to each state. With task discount factor gamma, the standard potential-based shaping term for a transition from s to s' is gamma times Phi(s') minus Phi(s). This term is added to the original reward.
Across a discounted trajectory, the intermediate potential terms cancel. For an episode ending after T transitions, the added discounted total is -Phi(s0) + gamma raised to T times Phi(sT). Setting terminal potentials to zero removes dependence on the ending state and duration. For a continuing discounted task, bounded potentials make the remaining limiting term vanish.
The resulting return shift depends only on the starting state, so it does not change which policy is preferred from that state under these conditions. This does not guarantee faster learning, reliable function approximation, or correct handling of truncated episodes. The potential, discount, and terminal convention must be implemented consistently.

Three practical examples

Simulated navigation: use a potential derived from estimated distance while retaining the actual destination reward.
Puzzle learning: provide a structured progress signal without independently rewarding every intermediate configuration.
Scheduling research: test whether a carefully defined potential helps learning from delayed completion feedback.

Think of it this way

Think of adding temporary bookkeeping credits as a task progresses, then reconciling them at the end. The intermediate statements may make progress easier to see, but a consistent reconciliation prevents the bookkeeping system from creating a better final score simply by moving back and forth.

Worked example

Consider a fictional task with discount factor 0.90 and terminal state G. Set potentials Phi(S) = -4, Phi(M) = -1, and Phi(G) = 0. One available path is S to M to G, with original rewards 0 and 10.
The first shaping reward is 0.90 times (-1) minus (-4) = 3.1. The second is 0.90 times 0 minus (-1) = 1. The shaped rewards are therefore 3.1 and 11. Original discounted return is 0 + 0.90 times 10 = 9. Shaped return is 3.1 + 0.90 times 11 = 13, exactly 4 higher.
A competing direct path S to G gives original reward 8. Its shaping reward is 0 - (-4) = 4, so its shaped return is 12. The two-step path remains preferable: 9 exceeds 8 originally, and 13 exceeds 12 after shaping.
Now insert an avoidable loop S to M to S before taking the direct path to G. Assume the loop transitions have original reward 0. The shaping terms are 3.1, then 0.90 times (-4) minus (-1) = -2.6, then 4. Their discounted sum is 3.1 + 0.90 times (-2.6) + 0.81 times 4 = 4. The original return is 0.81 times 8 = 6.48, and the shaped return is 10.48. The loop does not create an advantage over the direct path's 12.
For contrast, an arbitrary bonus of 3 for entering M, with no corresponding adjustment elsewhere, adds 3 to this looped path. Its resulting return is 3 + 0.81 times 8 = 9.48, above the direct path's 8. That different bonus rule changes the ranking.
These calculations use a finite, explicitly terminating task with zero terminal potential. Stopping a continuing simulation early while discarding its remaining potential term would require separate treatment; the cancellation argument cannot simply be assumed.

Illustrative case study

Fictional case: LindenNavigation gives a simulator agent points whenever it approaches a target. The agent learns repeated partial approaches because the implementation awards progress but does not account consistently for moving away. The team replaces the ad hoc bonus with a documented potential-based term and checks complete trajectory returns.
It reports both original task return and shaped training return, then compares learning curves across multiple runs. This invented case illustrates an incentive-design check, not evidence that one potential will improve every navigation problem.

Common misconceptions and mistakes

Common mistakes include calling any intermediate reward policy-preserving, omitting the discount factor from the potential difference, and assigning arbitrary nonzero terminal potentials when episodes stop. Another mistake is comparing shaped scores across different potentials as if they were the same business metric. A theoretically valid transformation can still interact poorly with finite training, approximation, or implementation errors.

Business use

Businesses can use shaping experiments to reduce the difficulty of learning from delayed outcomes, especially in simulation. Keep the original objective visible and evaluate final behavior against it. Documentation should identify the potential, discount convention, terminal handling, and observed learning benefit. If shaping changes the underlying objective intentionally, describe that as an objective redesign rather than claiming invariance.

Check your understanding

How much does valid shaping add to each complete return from S?

Answer: Four points in this example.

Why does the direct path score 12?

Answer: Its original 8 receives a shaping adjustment of 4.

Why is the loop not profitable under the valid construction?

Answer: The discounted potential adjustments reconcile to the same starting-state shift.

Related terms

Reward
Discount factor
Terminal states
potential function
policy invariance
sparse reward
reward hacking
terminal state
temporal-difference learning
value initialization

Learn next

Next, calculate shaping adjustments for several complete trajectories, including loops and different episode lengths. Verify the common return shift before testing whether the potential actually improves learning speed.

Sources

Andrew Y. Ng, Daishi Harada, and Stuart Russell, Policy Invariance under Reward Transformations: Theory and Application to Reward Shaping: https://people.eecs.berkeley.edu/~russell/papers/icml99-shaping.pdf Eric Wiewiora, Potential-Based Shaping and Q-Value Initialization are Equivalent: https://arxiv.org/abs/1106.5267

Previous: Reward hypothesis Next: Reward-to-go