Must behavior and target policies be completely different?

No. They can overlap substantially and differ only in some decisions.

Is a replay buffer required?

No. It is a common data-reuse mechanism, not the definition.

Can off-policy methods collect new data?

Yes. Fully offline training is the more restrictive case that does not collect additional environment experience during training.

AI Glossary / Reinforcement learning and robotics / Advanced

Off-policy

Definition

Off-policy describes reinforcement learning or policy evaluation that learns about a target policy using experience generated by a different behavior policy. The target policy is the one being evaluated or improved; the behavior policy is the one that selected the recorded actions. Keeping these roles separate allows useful experience to be reused, but requires methods that account appropriately for the mismatch.

What it means

A robot may explore several movements while the learning system estimates how a more focused policy would perform. Similarly, an analyst may evaluate a proposed decision rule using records produced by an earlier rule. The actions in the records need not have been selected by the policy of interest.
The distinction concerns the relationship between policies, not the age or storage location of the data. Off-policy learning can occur while new experience is still being collected. It is not automatically offline learning, and it does not mean any arbitrary historical dataset is sufficient.

How it works

First identify the behavior policy and target policy. A useful dataset records states or observations, chosen actions, rewards, subsequent observations, and information needed by the learning method. Whether the recorded state contains the relevant decision information is a substantive modeling question.
Some evaluation methods use importance sampling. An action receives a weight equal to its probability under the target policy divided by its probability under the behavior policy. This changes how behavior-generated outcomes contribute to a target-policy estimate. For sequential trajectories, products of ratios can create very large variability.
Other methods learn value functions through off-policy updates. Q-learning, for example, uses a target based on the highest estimated next-action value rather than necessarily the action the behavior policy actually chooses next. Not every off-policy method explicitly computes importance weights.
Coverage matters: if the target policy relies on actions absent from the behavior data in relevant states, their consequences cannot simply be recovered by reweighting. Function approximation and bootstrapping can introduce additional instability, so data reuse must be paired with an appropriate algorithm and evaluation.

Three practical examples

Robot learning: reuse transitions collected by earlier exploratory controllers while improving a new controller.
Service routing: estimate a proposed routing rule from logged decisions when the required action probabilities and relevant outcomes are available.
Simulation: learn values for a greedy decision rule while a separate exploratory rule continues trying alternative actions.

Think of it this way

Think of reviewing travel diaries written by people who chose routes differently from you. Their experiences may help estimate your proposed travel plan, but a simple average describes their choices, not yours. If nobody visited a destination you would frequently choose, relabeling their diaries cannot create the missing experience.

Worked example

Consider a one-step decision with two actions and no changing context. The behavior policy chooses A with probability 0.80 and B with probability 0.20. The target policy instead chooses A with probability 0.25 and B with probability 0.75. Assume deterministic rewards: A gives 2 units and B gives 8 units.
The behavior policy's expected reward is 0.80 × 2 + 0.20 × 8 = 3.2. The target policy's value is 0.25 × 2 + 0.75 × 8 = 6.5. Averaging behavior rewards without correction therefore answers the wrong policy question.
Importance weights are 0.25/0.80 = 0.3125 for A and 0.75/0.20 = 3.75 for B. Suppose a particular ten-decision dataset contains eight A outcomes and two B outcomes. Its ordinary importance-sampling estimate is [8 × 0.3125 × 2 + 2 × 3.75 × 8]/10 = 6.5.
This dataset happens to match the behavior proportions exactly. Another ten-decision dataset can give a different estimate even with deterministic action rewards because its action counts vary. An unbiased estimator need not equal the target value in every finite dataset.
If behavior instead always chose A, the probability of B would be zero and the B ratio would be undefined. The observed A rewards alone would not reveal the target policy's value without further assumptions or information about B.

Illustrative case study

Fictional case: AlderRouting wants to assess a new ticket-routing policy before testing it operationally. Its logs contain the previous policy's selected team and final resolution score, but initially omit selection probabilities. The analysts determine which evaluation methods the available records can support rather than inventing the missing probabilities.
They also find that some ticket categories were always routed to one team. They explicitly mark the proposed alternative routes in those categories as unsupported by direct comparisons in the logs. This fictional case illustrates evidence limitations, not a successful deployment or a claimed performance gain.

Common misconceptions and mistakes

Off-policy does not mean learning without rewards or ignoring the policy that collected data. Data being recent does not make it on-policy. Importance weighting does not fix zero coverage, unrecorded decision factors, or an unsuitable reward definition. A high offline estimate can also reflect estimation error or selective model choice rather than a genuinely better policy.

Business use

The practical attraction is reuse of expensive experience and separation of exploration from the policy being learned. Before relying on an estimate, establish what decisions the data represent, whether important target actions are covered, and how uncertainty is quantified. Compare different evaluation approaches where suitable and reserve independent evidence for final assessment. Reusing records may reduce collection cost, but it does not remove the need to validate the learned behavior.

Check your understanding

Which policy generated the worked dataset?

Answer: The behavior policy.

Why is B weighted more heavily?

Answer: The target chooses B more frequently than the behavior policy.

Why is the unweighted value 3.2 unsuitable for the target?

Answer: It averages outcomes under the behavior action proportions rather than the target proportions.

Related terms

On-policy
Importance sampling
Experience replay
Q-learning
Offline reinforcement learning
Behavior policy
target policy
coverage
off-policy evaluation
distribution shift

Learn next

Next, compare ordinary and normalized importance-sampling estimators. Then examine how long trajectories affect estimator variance and why off-policy control differs from evaluating a fixed policy.

Importance sampling

Sources

OpenAI Spinning Up, Algorithms, off-policy learning overview: https://spinningup.openai.com/en/latest/user/algorithms.html Sergey Levine and colleagues, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, off-policy distinctions and importance-sampling evaluation: https://arxiv.org/pdf/2005.01643

Independent review checked the tutorial’s importance-sampling equations and policy distinctions.

Previous: Off-line SPC Next: Offline inference