Is Q-learning the same as deep Q-learning?

No. Deep Q-learning uses a neural network to approximate Q-values and usually adds stabilizing mechanisms.

Why is it off-policy?

Its greedy learning target can differ from the exploratory behavior generating the transition.

AI Glossary / Reinforcement learning and robotics / Intermediate

Q-learning

Definition

Q-learning is a model-free reinforcement learning method that updates estimates of the long-term value of taking actions in states. Its update combines an observed reward with the highest estimated action value at the next state. It is off-policy because this target can describe a greedy continuation even when the behavior used to collect experience chooses exploratory actions.

What it means

The learner does not need a complete transition model before it begins. It can use observed transitions to revise its action values, often called Q-values. These estimates then help choose actions, while an exploration rule provides opportunities to learn about alternatives.
A Q-value is not simply an immediate reward or an action probability. It represents an estimate of discounted return after an action, with the learning objective directed toward optimal continuation. Early estimates can be wrong, and choosing actions from them without adequate exploration can leave useful options undiscovered.

How it works

For a transition from state S through action A to nonterminal state S', with reward R, the basic update is Q(S,A) ← Q(S,A) + α[R + γ max_b Q(S',b) − Q(S,A)]. The learning rate α controls how much the estimate changes, and γ controls the contribution of later rewards.
The bracketed difference is a temporal-difference error: it compares the current estimate with a new target. Because that target includes another learned estimate, Q-learning bootstraps. At a true terminal state, the future contribution is zero.
The behavior policy may choose actions through epsilon-greedy exploration or another suitable rule. The next action actually selected need not maximize Q, but the Q-learning target still uses the maximum over permitted next actions. This separates the learning target from the data-collection behavior.
Classical convergence results apply to finite tabular problems under conditions such as sufficient repeated state-action sampling, bounded rewards, appropriate discounting or termination assumptions, and suitable step-size schedules. They do not automatically extend to neural-network approximation or arbitrary fixed historical datasets. Those settings introduce additional issues such as instability, coverage gaps, and overestimation.

Three practical examples

Small simulated navigation: learn values for permitted movement actions from observed rewards and resulting locations.
Discrete resource experiments: compare actions in a manageable state space when the transition model is not supplied.
Educational benchmarks: investigate how exploration, reward design, and terminal handling affect learned behavior.

Think of it this way

Think of maintaining a notebook that rates decisions. After trying one, you revise its rating using what happened immediately and your current estimate of the best next opportunity. The notebook can improve through experience, but an overly optimistic rating elsewhere can also influence the current revision.

Worked example

Consider a fictional learner with current Q(S, Move) = 3. Set α = 0.25 and γ = 0.90. Taking Move gives reward −1 and reaches a nonterminal state T. The current values of its two permitted actions are 6 and 2.
The Q-learning target is −1 + 0.90 × max(6, 2) = 4.4. The temporal-difference error is 4.4 − 3 = 1.4. Updating one-quarter of that difference gives Q_new(S, Move) = 3 + 0.25 × 1.4 = 3.35.
Suppose the behavior policy then explores the action valued at 2. That does not change the Q-learning target just computed. A sampled SARSA update using that actual next action would instead target −1 + 0.90 × 2 = 0.8 and would update the original value 3 to 2.45. The difference reflects distinct continuation targets, not an arithmetic disagreement.
Now suppose a later observation of the same state-action pair has a different stochastic outcome: reward 10 followed by genuine termination. Starting from the updated value 3.35, the target is exactly 10. The next Q-learning update is 3.35 + 0.25 × (10 − 3.35) = 5.0125.
There is no future action value to add after termination. Incorrectly adding 0.90 × 6 would create target 15.4 and update the value to 6.3625, an inflated result for this observation. A time-limit cutoff that does not represent true task termination needs separate treatment based on the task definition.
These two updates illustrate mechanics only. They do not establish convergence or the true optimal value from two samples.

Illustrative case study

Fictional case: MapleNavigation trains a small simulated agent and finds unexpectedly high values near episode endings. Its team discovers that the update mistakenly carries a next-state value across genuine termination. It corrects the terminal handling, reruns the experiment, and compares actual returns as well as learned values.
The team separately records when episodes end because of a task outcome and when they stop because of an evaluation time limit. This fictional case illustrates a common modeling distinction and a diagnostic process. It does not report measured improvements for a real robot.

Common misconceptions and mistakes

Q-learning does not require every action to be chosen greedily during training. Model-free does not mean assumption-free. A larger Q-value estimate is not necessarily more accurate, and maximizing noisy estimates can produce overestimation. A table-based convergence theorem does not promise that every deep Q-learning implementation converges. Repeatedly replaying a narrow dataset also cannot supply missing experience for unsupported decisions.

Business use

This method is useful for understanding and testing sequential decision learning with discrete actions. Before using it operationally, assess state representation, reward quality, exploration cost, and coverage. Compare against simple baselines and evaluate the intended deployment policy under representative conditions. Keep claims about empirical performance separate from mathematical guarantees that apply only under specified assumptions.

Check your understanding

Which next-state value enters the first target?

Answer: Six, the maximum permitted value at T.

What is the first updated estimate?

Answer: 3.35.

Why is the later terminal target only 10?

Answer: The task has ended, so its future continuation value is zero.

Related terms

Off-policy
SARSA
Exploration
Deep Q-Network
Tabular Q-learning
Terminal states
Action-value function
temporal-difference learning
Bellman optimality equation
bootstrapping
terminal state

Learn next

Next, implement a small table-based example and inspect each update. Then examine maximization bias and compare ordinary Q-learning with Double Q-learning under noisy value estimates.

Double Q-learning

Sources

Christopher J. C. H. Watkins and Peter Dayan, Q-learning, original research and convergence statement: https://www.gatsby.ucl.ac.uk/~dayan/papers/wd92.html Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second edition, Section 6.5: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf

Previous: Q-function Next: Q-Q plot