Can passive learning happen online?

Yes. Estimates can update as new transitions arrive while the policy remains fixed.

Is it necessarily supervised learning?

No. The reinforcement-learning setting uses rewards and state transitions to estimate returns or dynamics, rather than requiring a correct action label for every state.

Can the fixed policy be randomized?

Yes. A fixed stochastic rule can assign probabilities to actions; individual choices can vary while the policy itself remains unchanged.

AI Glossary / Graphs and symbolic AI / Intermediate

Passive learning

Definition

Passive learning, in the reinforcement-learning sense used here, means learning about the value or consequences of following a fixed policy. The agent gathers experience under that policy rather than choosing actions to improve or explore alternatives during the learning task.

What it means

A policy specifies behavior in each relevant situation. If a robot always follows a prescribed route rule, passive learning can estimate the expected reward from different locations under that rule. It does not automatically discover a better route rule.

The fixed object is the policy, not necessarily the learned model or value estimates. These estimates can change substantially as new experience arrives. The policy may be deterministic or stochastic, provided its action-selection rule remains fixed for the task being evaluated.

Outside reinforcement learning, passive learning often contrasts receiving supplied examples with actively selecting queries or labels. That related usage should be distinguished from the fixed-policy evaluation focus of this entry.

How it works

The learner observes states, actions, rewards and subsequent states generated while following the prescribed policy. It may estimate a transition-and-reward model and then evaluate the policy using that model.

Alternatively, Monte Carlo methods average observed returns from completed episodes, while temporal-difference methods update estimates using an immediate reward and an estimate of the next state's value. A simple TD(0) update is V(s) ← V(s) + alpha[r + gamma V(s') − V(s)], with a suitable terminal-state convention.

The target is the expected return under the policy, not the best achievable return over all possible policies. Learning about unvisited states or actions requires additional assumptions or evidence; fixed behavior can leave important alternatives unexplored.

Care is also needed when evaluating a policy from data generated by a different policy. That is an off-policy evaluation problem, not automatically solved by averaging the available outcomes as though they came from the target policy.

Three practical examples

A simulated robot estimates the reward associated with a prescribed navigation rule before the rule is revised.
A research team uses temporal-difference updates to learn state values while actions remain governed by a fixed policy.
An operations model learns the consequences of a specified dispatch routine without letting the learner alter dispatch decisions during the evaluation.

Think of it this way

Think of learning how long a fixed bus route usually takes by repeatedly riding it. Your estimates improve as you observe more journeys, but you are not choosing alternative roads to find the fastest route.

The analogy is limited because reinforcement-learning values can combine many rewards and discount future outcomes. Also, the environment may be stochastic. “Fixed policy” means the decision rule stays fixed; it does not require every journey or every chosen action to be identical.

Worked example

Consider an invented episodic system with states S, T and a terminal state End. Under a fixed policy, S always leads to T with reward 0, and T leads to End with reward 10. Let gamma=0.9 and terminal value V(End)=0.

The true policy values are V(T)=10 and V(S)=0 + 0.9×10=9. These describe this policy's expected returns in the deterministic toy system.

Initialize V(S)=V(T)=0 and use alpha=0.5. On the first episode, the S→T update leaves V(S)=0 because the current estimate of V(T) is zero. The T→End update gives V(T)=0 + 0.5×10=5.

On the next S→T transition, V(S) becomes 0 + 0.5×(0.9×5)=2.25. Estimates propagate information across visits even though the policy never changes. This arithmetic demonstrates individual updates, not a general convergence guarantee for a constant learning rate in noisy environments.

Illustrative case study

The fictional Alder Routes team evaluates a fixed simulated routing policy. It sees improving value estimates and initially describes the agent as learning better actions, even though the action-selection rule has remained unchanged.

The team corrects its report to describe policy evaluation. When it later introduces a policy-improvement step, it records that as a different phase and reevaluates the resulting behavior.

This fictional case illustrates why improved knowledge and improved behavior should be measured separately. A more accurate estimate can reveal that an unchanged policy performs poorly without itself changing any action.

Common misconceptions and mistakes

Passive means the learner does no computation. It can perform substantial model fitting and value estimation.
A fixed policy means deterministic outcomes. Transitions and rewards can remain stochastic.
Better value estimates imply better behavior. Behavior may be identical while its evaluation becomes more accurate.
Passive learning identifies the best untried action. Evidence under a fixed policy may not support that comparison.
Any logged dataset directly evaluates any policy. Differences between behavior and target policies require appropriate treatment.

Business use

Passive learning supports assessment of an existing decision rule before considering changes. It can be useful when a project wants to separate estimating current behavior from optimizing future behavior.

Specify the policy, reward definition, discount factor, episode boundaries and observation process. Explain which states are adequately covered and which remain poorly supported. If conditions change, updated estimates may reflect a changed environment rather than a changed policy.

When reporting results, distinguish the estimated value of the current policy from claims about alternatives. A business can use the evaluation to decide whether improvement is needed, but selecting a better policy requires an additional control or comparison step.

Check your understanding

Why does V(S) remain zero after the first transition in the worked example?

Answer: The immediate reward is zero and the current estimate of V(T) is still zero. The later reward first updates T's estimate; a subsequent transition allows that information to influence S.

Related terms

Policy evaluation
Reinforcement learning
Adaptive dynamic programming
temporal-difference learning
Monte Carlo return
fixed policy
on-policy learning
off-policy evaluation

Learn next

Choose a short fixed-policy episode and compute its discounted return. Then perform TD updates from zero initial values, noting when later rewards influence earlier states. Finally, identify what additional work would be required to compare or improve the policy.

Sources

Carnegie Mellon reinforcement-learning notes distinguish passive fixed-policy learning from active control and explain value updates: https://www.cs.cmu.edu/~15281-f23/coursenotes/reinforcement/index.html

Sutton and Barto develop policy evaluation; search excerpts were available, but the PDF could not be opened during review: https://www.incompleteideas.net/book/bookdraft2018mar21.pdf

The two-state calculation and Alder Routes case are original examples in the reinforcement-learning sense.

Previous: Passive-aggressive learning Next: Password fatigue