Can an on-policy method use a value function?

Yes. SARSA and actor-critic methods can do so.

Is on-policy always more stable?

No. Stability depends on the algorithm, approximation, optimization, and environment.

Does it guarantee an optimal policy?

No. Convergence claims require specific assumptions, and practical function approximation can fall outside those conditions.

AI Glossary / Reinforcement learning and robotics / Intermediate

On-policy

Definition

On-policy describes reinforcement learning in which the policy being evaluated or improved is also the policy used to generate the relevant experience. Learning therefore reflects the behavior the agent is actually following, including its exploratory choices. The term concerns the relationship between learning and data collection; it does not imply that the policy is already good, deterministic, or optimal.

What it means

An agent can learn the consequences of its current way of acting and then adjust that behavior. If the current policy sometimes explores, those exploratory actions are part of the behavior whose value is being learned. The learner should not silently replace their consequences with those of an idealized policy that always chooses the apparent best action.
This relationship changes as the policy changes. Experience collected under an earlier policy may no longer represent the new policy's decision distribution. On-policy methods therefore commonly gather fresh experience during training, although the exact rules for reusing a recent batch depend on the algorithm.

How it works

A basic learning cycle selects actions under the current policy, records outcomes, estimates values or policy improvements from those outcomes, and updates the policy. The relevant state and action distributions should match the learning objective.
SARSA provides a concrete example. After state S and action A produce reward R and next state S', the policy selects the next action A'. Its one-step update uses the target R + γQ(S', A'), with no future term when the episode has truly terminated. This incorporates the next action actually selected by the policy.
By contrast, a Q-learning target uses the maximum estimated next-action value regardless of the exploratory action taken next. That difference explains how similar-looking value updates can have different policy relationships.
On-policy policy-gradient methods estimate improvements using trajectories associated with the policy being optimized. Practical algorithms may perform multiple updates on a recently collected batch using specific controls on policy change. This does not justify arbitrary reuse of unrelated historical experience. Matching the data source, objective, and update rule remains essential.

Three practical examples

Simulated navigation: learn the expected return of an exploratory movement policy while that same policy controls the agent.
Game playing: collect episodes under the current strategy and use their outcomes to adjust that strategy.
Robot-control research: evaluate a current controller's behavior in a simulator before collecting the next round of training trajectories.

Think of it this way

Think of practicing a route using your actual habits, including occasional exploratory turns. You want to learn what those habits lead to, not what a perfect navigator would achieve. Once you change your habits substantially, the old practice runs are no longer a direct picture of your new behavior.

Worked example

Suppose a SARSA learner currently estimates Q(S, A) = 5. Taking A gives reward 2 and leads to a nonterminal state T. At T, the current estimates are Q(T, Left) = 8 and Q(T, Right) = 1. Let the learning rate be α = 0.20 and the discount factor γ = 0.90.
The current policy selects Left with probability 0.75 and Right with probability 0.25. On this particular transition it selects Right. The SARSA target is 2 + 0.90 × 1 = 2.9. The updated estimate is 5 + 0.20 × (2.9 − 5) = 4.58.
If the policy had selected Left instead, the target would be 2 + 0.90 × 8 = 9.2 and the update would give 5.84. Both are valid sampled SARSA updates under the stated stochastic policy.
Averaging over that next-action choice gives expected target 0.75 × 9.2 + 0.25 × 2.9 = 7.625. The expected updated value is therefore 5 + 0.20 × (7.625 − 5) = 5.525. An Expected SARSA update using this same policy distribution would directly use that averaged target.
A Q-learning update would use the maximum next value 8 and produce 5.84 regardless of whether Right was selected. The contrast is about which policy's continuation is represented, not about one sampled update proving that either algorithm is better.

Illustrative case study

Fictional case: HazelGames trains a simulated game agent using a stochastic strategy. Its team notices that an evaluation dashboard reports results from a greedy version of the policy while training records include exploration. The two numbers answer different questions, so the team labels them separately and records which action rule produced each run.
When changing the training strategy, it collects the data required by its chosen update method rather than assuming all older episodes remain interchangeable. This fictional case illustrates policy consistency and reporting discipline. It does not report a real training result or promise that on-policy training will be more successful.

Common misconceptions and mistakes

On-policy does not mean always choosing the action with the highest estimated value. Exploration can be part of the policy. It is not synonymous with online computing, and data being freshly collected does not by itself make a learning rule on-policy. Conversely, multiple optimization passes over a recent batch do not automatically make an algorithm conventionally off-policy; the objective and treatment of policy changes matter.

Business use

On-policy approaches can offer a direct connection between the behavior being improved and the experience used for that improvement. Their need for suitable new experience can make data collection a substantial cost. Compare approaches using environment interactions, computation, and actual performance under stated evaluation conditions. Keep the policy version and exploration settings alongside each dataset so reported results can be interpreted correctly.

Check your understanding

Which next-action value enters the observed SARSA update?

Answer: Q(T, Right) = 1, because Right was selected.

Why is the update smaller than the Q-learning update?

Answer: It incorporates the lower-valued exploratory continuation.

What does the expected target 7.625 average over?

Answer: The current policy's probabilities for Left and Right at T.

Related terms

Off-policy
SARSA
Expected SARSA
Policy gradient
Exploration
Q-learning
behavior policy
target policy
temporal-difference learning
actor-critic

Learn next

Next, compare sampled SARSA with Expected SARSA on the same policy. Then study how policy improvement changes the experience needed for evaluation and why training and deployment action rules should be distinguished.

Sources

Stanford CS234, Lecture 4 notes, on-policy SARSA and control: https://web.stanford.edu/class/archive/cs/cs234/cs234.1224/CS234Win2019/slides/lnotes4.pdf OpenAI Spinning Up, Algorithms, on-policy and off-policy families: https://spinningup.openai.com/en/latest/user/algorithms.html

Independent review checked SARSA and Q-learning update equations.

Previous: On-line SPC Next: One-armed bandit