Can the learned policy outperform the recorded behavior?

Sometimes, when the data and assumptions support useful improvement; it is not guaranteed.

Does a large dataset remove distribution shift?

No. It can still omit important state-action combinations.

Is offline evaluation the same as offline learning?

No. Evaluation estimates a policy's performance; learning selects or changes the policy.

AI Glossary / Reinforcement learning and robotics / Advanced

Offline reinforcement learning

Definition

Offline reinforcement learning is the process of learning a decision-making policy from an existing dataset of experience without collecting additional environment interactions during training. The records typically describe states or observations, actions, rewards, and subsequent outcomes. The aim is to learn useful sequential behavior from available evidence, while accounting for the limits and biases of that evidence.

What it means

An organization may already have many records of decisions and their consequences, yet be unable to let a learning system freely experiment. Offline reinforcement learning asks whether those records can support a better policy before any new operational trial.
The word offline refers to the training interaction setting. It does not mean that the computer must be disconnected from the internet. It also does not mean the final policy is never deployed. A later phase that collects new experience and updates the policy is commonly described as online fine-tuning or offline-to-online learning.

How it works

The training dataset is fixed for the offline learning phase. It may contain experience from several previous policies, human decisions, or other sources. The learner uses those records to estimate values, learn a model, or directly improve a policy under a reward objective.
A central difficulty is that the learned policy may choose actions or encounter states that are poorly represented in the dataset. A value model can assign an unsupported action an unrealistically high value. Optimization may then favor that error, and sequential value updates can propagate it backward to earlier decisions.
Methods address this problem in different ways. Some constrain the policy toward behavior supported by the data. Some use conservative value estimates or model uncertainty. These choices involve tradeoffs: excessive restriction can prevent improvement, while insufficient restriction can encourage unsupported behavior.
Evaluation is a separate challenge. A low training loss is not proof of effective control. Appropriate off-policy evaluation, careful data separation, model checks, and eventual external validation may all contribute, depending on what evidence is available.

Three practical examples

Robot manipulation: learn from recorded arm movements and outcomes when further physical trials are expensive.
Resource scheduling: investigate a new sequential allocation policy using historical demand, decisions, costs, and subsequent system states.
Interactive assistance: study dialogue policies using logged exchanges and meaningful outcome signals, while checking whether the logs capture information needed to interpret past choices.

Think of it this way

Think of preparing for an unfamiliar route using a fixed collection of travel journals. You can compare recorded journeys and combine compatible information, but you cannot ask the journals what happened on an unvisited road. A polished map drawn from those journals can still be wrong where the evidence is thin.

Worked example

Consider a fictional two-step system with discount factor γ = 0.90. At state S, action Stop ends the episode and gives reward 4. Action Continue gives reward 0 and moves to state T. The dataset includes examples of both choices at S. At T, however, it contains only action Safe, which ends the episode with reward 3.
Using only these supported choices, Continue has value 0 + 0.90 × 3 = 2.7, so Stop is preferred to Continue. Now suppose a fitted value model assigns value 20 to another action, Fast, at T even though Fast never appears in the dataset.
An unconstrained maximizing backup uses the larger predicted value and gives Continue a target of 0 + 0.90 × 20 = 18. The learner may then prefer Continue over Stop because of an unsupported prediction, despite fitting every recorded Safe outcome correctly.
To expose the evidence gap, consider two possible environments consistent with all the logged data. In environment One, Fast gives reward −10, making Continue followed by Fast worth −9. In environment Two, Fast gives reward 30, making that sequence worth 27. Both environments produce exactly the same recorded observations because the logs never contain Fast.
The dataset alone cannot distinguish these environments. Restricting the example's choices at T to Safe produces the supported value 2.7, but it does not prove that Fast is bad in reality. This is a demonstration of missing information, not a derivation of a particular offline learning algorithm or a universal rule to discard every unobserved action.

Illustrative case study

Fictional case: MapleScheduling assembles a dataset of machine assignments and later completion times. The team discovers that a particular machine was used only for simple jobs, while a proposed policy would assign it difficult jobs as well. A model's favorable prediction for that new combination is therefore not directly supported by the historical assignments.
The team records this gap, compares policies within the evidence available, and separates model selection from its final assessment. This fictional case illustrates why many rows do not necessarily provide broad decision coverage. It makes no claim that offline learning automatically improves scheduling.

Common misconceptions and mistakes

Offline reinforcement learning is not simply behavior cloning: cloning primarily imitates recorded actions, whereas offline RL optimizes a reward-based objective. It is also not equivalent to ordinary off-policy learning with an actively growing replay buffer. Repeating optimization on the same records does not create new environmental evidence. Synthetic rollouts inherit the assumptions and errors of the model that generated them.

Business use

This approach can make existing operational records useful for investigating sequential decision policies. Begin by checking reward quality, state information, action coverage, and whether the historical process resembles the intended use. Evaluate against simple baselines, including imitation where appropriate. Preserve the boundaries of what the data can establish, especially when the proposed policy changes which situations the system will encounter.

Check your understanding

Why does Continue appear worth 18 in the example?

Answer: Its backup uses an unsupported predicted value of 20 at the next state.

Can the logs tell whether Fast yields −10 or 30?

Answer: No, because Fast was never recorded.

What happens if new environment data are collected during training?

Answer: The process has moved beyond the strictly offline phase.

Related terms

Conservative Q-learning
Replay buffer
Model-based reinforcement learning
Off-policy learning
behavior cloning
dataset coverage
distribution shift
behavior policy
off-policy evaluation

Learn next

Next, compare policy constraints with conservative value estimation. Then examine how evaluation changes when behavior probabilities are known, unknown, or zero for actions of interest.

Sources

Sergey Levine and colleagues, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems: https://arxiv.org/pdf/2005.01643 Aviral Kumar and colleagues, Conservative Q-Learning for Offline Reinforcement Learning, NeurIPS 2020: https://papers.nips.cc/paper/2020/hash/0d2b2061826a5df3221116a5085a6052-Abstract.html Berkeley AI Research, Offline Reinforcement Learning: How Conservative Algorithms Can Enable New Applications: https://bairblog.github.io/2020/12/07/offline/

Previous: Offline private key protocol Next: Offline search