Must the model be a neural network?

No. It can be a table, simulator, structured equation, probabilistic model, or learned representation.

Does every action require a new search?

No. Some methods use the model to train a policy that later acts directly.

Is model-based reinforcement learning always more sample efficient?

No. Its advantage depends on model quality, task structure, and how predictions are used.

AI Glossary / Reinforcement learning and robotics / Advanced

Model-based reinforcement learning

Definition

Model-based reinforcement learning uses a model of an environment's consequences to help an agent learn or choose actions that maximize expected cumulative reward. The model predicts outcomes such as next states and rewards. It may be learned from experience or supplied, and can support planning, imagined training experience, or both.

What it means

An agent can improve decisions by asking what would happen after possible actions before carrying them out. Model-based reinforcement learning makes that prediction process an explicit part of learning or decision-making.
The word model refers to a model of the environment's behavior, not simply any neural network in the agent. A neural policy or value function alone does not make an algorithm model based. The distinction concerns how predicted consequences are used to improve behavior.

How it works

A common workflow collects transitions containing a state, action, reward, and next state. A predictive model is fitted to those observations. The agent then uses predictions to evaluate action sequences or generate synthetic transitions for additional policy or value updates.
Dyna-style methods interleave learning from real experience with learning from model-generated experience. Other approaches plan repeatedly at decision time, execute an action, observe the actual outcome, and plan again. Models may predict full observations or a learned latent state sufficient for the intended decisions.
The central difficulty is model error. A policy can seek actions where a learned predictor is overly optimistic, and long imagined sequences can compound prediction errors. Shorter rollouts, uncertainty handling, and frequent correction from real observations are possible responses, but none universally eliminates the problem.
The reward specification and planning horizon also matter. Accurate prediction of irrelevant details does not guarantee that the model preserves the consequences needed for a good decision.

Three practical examples

Robot learning: a dynamics model predicts the effect of candidate movements so a learner can use limited physical interaction more efficiently.
Resource allocation simulation: a learned transition model supports exploration of sequential operating decisions before further evaluation in the real process.
Game agents: an environment simulator or learned predictive model supports lookahead and learning about future consequences.

Think of it this way

Think of practicing a sequence on a rehearsal model before acting in the real setting. Rehearsal can make each real attempt more informative, but a misleading rehearsal environment can teach the wrong response. The agent must distinguish what the model predicts from what observations have established, especially when it explores unfamiliar situations.

Worked example

Consider an original two-step decision problem. At the initial state, the agent chooses Fast or Steady. After that action, one final reward is received and the episode ends. Use discount factor gamma = 0.9 for that final reward.
The learned model predicts that Fast gives immediate reward 1, then reaches a state paying final reward 10 with probability 0.8 and a state paying 0 otherwise. Its predicted return is 1 + 0.9 times (0.8 times 10 + 0.2 times 0) = 8.2.
Steady gives immediate reward 2 and reaches a state paying final reward 6 with certainty. Its predicted return is 2 + 0.9 times 6 = 7.4. Planning with these predictions selects Fast.
Suppose further evaluation reveals that Fast's true probability of reaching the high-reward state is 0.5. Its actual expected return is then 1 + 0.9 times 5 = 5.5, below Steady's 7.4. The optimizer performed the model calculation correctly; the transition estimate was wrong.
More generally, if Fast's success probability is p, it is preferred only when 1 + 9p > 7.4, or p > 32/45, approximately 0.7111. An uncertainty range spanning that threshold leaves the action ranking unresolved.
The example isolates one transition error with exact reward values and a short horizon. A real learned model may also mispredict rewards, timing, or later states. Generating many more synthetic trajectories from the same incorrect probability would reduce simulation noise without correcting the underlying prediction error.

Illustrative case study

Fictional case: Cedar Automation develops a learning controller for a simulated packaging process. Its team fits a transition model from recorded trials and uses short imagined sequences to propose improved policies.
Before relying on a proposal, engineers test it on fresh trials that were not used to train the model. When the proposed policy visits an unfamiliar operating region, they compare prediction errors there with errors in familiar conditions and collect additional appropriate data.
They track real interaction cost separately from model-training and planning computation. This hypothetical case illustrates how a team might evaluate sample efficiency and model exploitation; it is not a report of measured production improvement.

Common misconceptions and mistakes

Model based does not mean the environment model is correct or that all uncertainty is known. More imagined data is not equivalent to more independent real evidence. A value network is not automatically a transition model.
Good one-step average prediction error can hide errors in states that matter to the policy. A short planning horizon may miss delayed effects, while a long horizon can amplify model errors. Finally, a policy optimized under the specified reward can still be undesirable if that reward omits important requirements.

Business use

The approach can be valuable when real interaction is expensive and useful predictive structure can be learned. Its assessment should include the cost of model training, planning latency, real-world validation, and response to changing conditions. Teams should compare total resources and decision quality against simpler methods rather than report only the number of physical trials saved. The model's operating range should be explicit.

Check your understanding

Why does generating more imagined Fast outcomes not necessarily correct the bad decision in the example?

Answer: If the model keeps using success probability 0.8, additional simulated outcomes converge toward the wrong expected return of 8.2. They do not reveal the true probability 0.5.

Related terms

Reinforcement learning
Model-free (reinforcement learning)
Environment model
Dyna
Planning
Model predictive control
World model
Model bias

Learn next

Learn Bellman backups with a small known transition table, then replace the true transitions with estimates from limited data. Compare planning decisions as those estimates change and examine how model errors affect action rankings.

Sources

Richard S. Sutton, Integrated Modeling and Control Based on Reinforcement Learning and Dynamic Programming, Dyna architectures: https://papers.neurips.cc/paper_files/paper/1990/file/d9fc5b73a8d78fad3d6dffe419384e70-Paper.pdf Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine, When to Trust Your Model: Model-Based Policy Optimization (2019): https://proceedings.neurips.cc/paper/2019/file/5faf461eff3099671ad63c6f3f094f7f-Paper.pdf

Independent review checked Dyna integration and MBPO model-error discussion.

Previous: Model-based reasoning Next: Model capacity