Can model-free learning use a neural network?

Yes. Networks can represent policies and values without representing transition dynamics for planning.

Can it train in simulation?

Yes. A simulator can provide experience to a model-free algorithm.

Is Q-learning always model-free?

Its direct update is model-free, but that update can also be used inside a larger architecture that performs model-based planning.

AI Glossary / Reinforcement learning and robotics / Intermediate

Model-free (reinforcement learning)

Definition

Model-free reinforcement learning learns value estimates or a policy from experience without using an explicit model of the environment's transition and reward dynamics for planning. The agent improves decisions using observed outcomes, such as rewards and subsequent states. Model-free does not mean that no mathematical model or neural network exists: a learned policy or value function can itself be a substantial model.

What it means

An agent can learn that an action tends to be useful without first constructing a predictor of everything that might happen next. It can update an estimate of long-term value from experience or adjust a policy to favor actions associated with higher returns.
The distinction concerns how the learning or decision process uses environmental dynamics. A dynamics model predicts consequences of actions for planning. A value function instead estimates expected return, and a policy specifies behavior. These objects serve different roles even when all are represented by neural networks.

How it works

Define states or observations, actions, rewards, and the return the agent should optimize. Collect experience through interaction or an existing dataset, subject to the method's assumptions and the operational setting.
A value-based method updates estimates from sampled rewards and subsequent observations. Q-learning, for example, updates an action value toward the observed reward plus a discounted estimate of the best next-state action value. Policy-based methods can instead adjust policy parameters using estimates of how actions affect return. Actor - critic methods combine policy and value components.
No explicit transition model is needed for these direct updates. However, model-free algorithms can be trained using a simulator that supplies experience. This does not automatically mean the algorithm performs model-based planning internally. Conversely, a larger architecture can combine direct model-free updates with a learned dynamics model and planning, as in Dyna.
Evaluate data efficiency, exploration, stability, and generalization. Off-policy learning does not make arbitrary historical data sufficient, and sparse or misleading rewards can obstruct progress. Convergence results for particular tabular algorithms should not be generalized without qualification to every neural-network implementation or deployment setting.

Three practical examples

A simulated navigation agent learns action values from trajectories without constructing its own map of transition probabilities for planning.
A controller learns a policy from repeated task outcomes while engineers separately assess whether the resulting behavior meets operating requirements.
A research team compares direct learning with a model-based method under the same interaction budget, measuring both returns and the amount of experience required.

Think of it this way

Think of learning which route usually gets you home quickly from repeated trips, without building a detailed traffic simulator. You maintain useful expectations about choices, but you do not explicitly predict every possible traffic transition before acting. The analogy describes the distinction, not a guarantee that experience alone will cover every future situation.

Worked example

Consider one invented tabular Q-learning update. The current estimate for taking action a in state s is Q(s, a) = 5. The observed reward is 2, the discount factor is 0.9, and the largest estimated action value in the next state is 6. Let the learning rate be 0.2.
The update target is 2 + 0.9 × 6 = 7.4. The temporal-difference error is 7.4 − 5 = 2.4. The new estimate is 5 + 0.2 × 2.4 = 5.48. This uses one observed transition and the current value estimates; it does not require an explicit table of transition probabilities.
For a separate terminal transition, suppose the old estimate is again 5 and the terminal reward is 2. With no future return after termination, the target is 2, so the update becomes 5 + 0.2 × (2 − 5) = 4.4.
The two results differ because a terminal state has no remaining return to bootstrap. One update does not establish the correct long-term value, and the fixed learning rate in this illustration is not a claim that all conditions for asymptotic convergence have been satisfied.

Illustrative case study

In this fictional case, Fir Warehouse Research trains a navigation policy in a simulator. Its model-free learner receives rewards and observations but does not build a transition model for look-ahead planning.
The team initially sees high training returns, then discovers poor performance on layouts excluded from training. It expands evaluation and compares the policy with a simpler navigation baseline. It also checks whether the reward unintentionally favors behavior that looks efficient in the simulator but fails the intended task.
The report distinguishes simulated experience, the learner's internal method, and physical deployment evidence. Successful training is treated as a candidate result requiring further validation, rather than proof of reliable real-world operation.

Common misconceptions and mistakes

Model-free does not mean assumption-free, representation-free, or data-free. Neural policies and value functions can contain many parameters even though they are not environment dynamics models.
Another mistake is equating model-free with online learning only. Some methods learn from stored experience, although dataset coverage and distribution shift create important challenges. Likewise, using simulated experience does not by itself settle whether the learning architecture is model-based.
Finally, a value estimate is not a prediction of the exact next state. It summarizes expected return under the relevant policy or optimality target and can be inaccurate.

Business use

Model-free methods can be useful when direct experience is available and constructing a useful dynamics model is difficult. The business assessment should include the cost of obtaining that experience, acceptable exploration, evaluation coverage, and the meaning of the reward.
Compare against simpler baselines and model-based alternatives. Keep training results separate from evidence about performance under changed conditions, and define how the deployed system is monitored and updated.

Check your understanding

What is the nonterminal example's target?

Answer: 7.4.

Why is the terminal update smaller?

Answer: Its target includes no future return beyond the terminal reward.

What distinguishes a value function from a dynamics model?

Answer: One estimates return; the other predicts environmental consequences of actions.

Related terms

Model-based reinforcement learning
Q-learning
Policy gradient
Exploration
Return
actor–critic
temporal-difference learning
Dyna

Learn next

Learn next how Q-learning and policy-gradient methods use experience. Compare them with model-based planning and examine the conditions behind tabular convergence results before extending those conclusions to function approximation.

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second edition, Sections 6.5 and 8.1–8.2: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf

Previous: Model extraction Next: Model Governance