Can the learned policy improve on historical behavior?

It can seek improvement, but improvement must be evaluated rather than assumed.

Does every method require an explicit behavior model?

No; estimation and constraint mechanisms differ across methods.

Is KL divergence symmetric?

No. Its direction affects which departures receive large penalties and must be stated.

AI Glossary / Probabilistic machine learning / Advanced

Behavior regularization in offline reinforcement learning

Definition

Behavior regularization in offline reinforcement learning encourages a learned decision policy to remain close to the actions represented in a fixed historical dataset. It limits departures from recorded behavior while seeking higher rewards, helping control reliance on action values that the available evidence supports poorly.

What it means

Imagine learning how to operate equipment from past operating logs, without experimenting on the equipment during training. A model might predict excellent results for settings rarely used in those logs. That prediction may reflect an unreliable extrapolation rather than a real opportunity.
Behavior regularization adds a reason to hesitate before selecting such settings. The reference behavior can be estimated from the dataset, including records collected by several historical policies. Staying near that reference does not mean the historical decisions were optimal, fair, or safe. It is a restriction motivated by limited evidence.

How it works

An actor proposes actions, while a critic estimates their future rewards. A behavior model can estimate the distribution of recorded actions conditional on the observed state. Training then balances predicted return against a measure of distance from that reference distribution.
One schematic objective is expected estimated action value minus alpha times a policy divergence. Alpha controls the trade-off, and its interpretation depends on reward scaling. KL divergence is one option; other approaches use different discrepancy measures or restrict the policy's available actions.
The BRAC framework distinguishes a penalty applied during policy improvement from a penalty also incorporated into value learning. These choices change the optimization problem. A behavior model's inaccurate probabilities can also distort the restriction, so fitting that model is part of the challenge, not a guarantee that historical support has been recovered.

Three practical examples

1. A warehouse analyst could learn dispatch choices from recorded shifts, discouraging an abrupt move toward poorly represented dispatch patterns before additional evaluation.
2. An energy researcher could compare operating policies using historical observations, examining whether apparent gains depend on actions with weak coverage in particular weather conditions.
3. A software team could study request-routing policies from service logs, separating higher predicted performance from how far a proposed policy departs from routing decisions previously observed.
These are possible analytical uses, not evidence that regularization alone makes deployment appropriate.

Think of it this way

Think of planning a walk from a map drawn only along previously traveled paths. A route beyond the mapped area might be shorter, but its apparent advantage depends on uncertain terrain. Behavior regularization charges for leaving familiar territory. It does not repair an inaccurate map or prove that the familiar paths are good ones.

Worked example

Consider an original, simplified one-state decision problem with two actions, A and B. Suppose the recorded behavior chooses A with probability 0.9 and B with probability 0.1. A fitted critic assigns values 2 and 6 respectively. Treat these as estimates, not verified rewards.
Compare policies using J = expected estimated value minus 4 times KL(policy || behavior), with natural logarithms. The behavior policy itself has estimated value 2.4, zero divergence, and J = 2.4.
A policy choosing each action with probability 0.5 has estimated value 4. Its KL divergence is approximately 0.510826, so J = 4 − 4 × 0.510826 = 1.956698. Despite its higher estimated reward, this policy scores worse after the penalty.
For this particular objective, the maximizing probabilities are proportional to behavior probability times exp(action value / 4). Normalizing gives A approximately 0.768031 and B approximately 0.231969. Its estimated value is 2.927877, divergence 0.073404, and penalized score 2.634260.
Thus the regularized optimum increases B's probability while retaining substantial weight on A. Without this penalty, maximizing the stipulated critic alone chooses B every time. None of these calculations establishes B's real-world value; the example isolates the effect of the objective in a one-step setting.

Illustrative case study

In a fictional scenario, Willow Dispatch has delivery logs from several past scheduling rules. Its analysts build an offline learning dataset and find that an unconstrained candidate often chooses routes that were uncommon in comparable operating conditions.
They fit a reference behavior model and compare several regularization strengths. For each candidate, they inspect coverage, estimated return, and departures from historical actions, including whether a small overall discrepancy hides large changes in individual operating situations.
The team retains its existing dispatch process while investigating the candidates. The fictional lesson is to make unsupported policy changes visible and evaluate them deliberately. No performance improvement or successful deployment is claimed.

Common misconceptions and mistakes

“Offline” does not simply mean a computer is disconnected from the internet; it means learning without collecting new environment interactions in this setting. Behavior regularization is not identical to behavior cloning: reward estimates still influence the learned policy. A larger penalty is not automatically better because excessive restriction can preserve poor historical choices. Small average divergence does not establish good behavior in every state. Regularizing actions also does not resolve missing state information, biased rewards, or changing operating conditions.

Business use

This concept helps teams question whether a proposed decision policy earns its predicted gains through credible evidence or unsupported extrapolation. Useful review materials include how the reference behavior was estimated, where historical coverage is weak, and how candidate choices change as the penalty changes. Evaluation should address the business outcome itself, not merely a higher regularized training score. Historical behavior may reflect constraints or biases that deserve separate examination.

Check your understanding

Why does the 50:50 policy lose to the reference in the example?

Answer: Its additional estimated value is outweighed by its divergence penalty.

Why is the regularized optimum still uncertain operationally?

Answer: It depends on critic estimates and a simplified setting.

What happens to the reward-only objective when alpha is zero?

Answer: The divergence term disappears, leaving the stated action-value maximization problem.

Related terms

Offline reinforcement learning
behavior policy
behavior cloning
actor-critic
distribution shift
Kullback–Leibler divergence
off-policy evaluation

Learn next

Compare policy constraints with conservative value estimation. Then study how offline policy evaluation depends on coverage and assumptions about the logged data.

Policy evaluation

Sources

Wu, Tucker, and Nachum, Behavior Regularized Offline Reinforcement Learning (2019), especially Sections 2–3: https://arxiv.org/abs/1911.11361 . The worked example and Willow Dispatch scenario are original illustrations.

Previous: Behavior-based robotics Next: Behavior tree