Does tabular Q-learning use a neural network?

No; the action values are stored explicitly.

Does one update change similar states?

Not in the basic tabular method.

Can it work with continuous inputs?

Inputs can be discretized, but that introduces approximation and state-design tradeoffs.

AI Glossary / Reinforcement learning and robotics / Intermediate

Tabular Q-learning

Definition

Tabular Q-learning is Q-learning implemented with an explicit table of action-value estimates, one entry for each represented state-action pair. The learner updates individual entries from observed transitions and rewards. It is useful when the state and action spaces are small enough to enumerate, and it makes the learning process easier to inspect than methods that approximate values with a neural network.

What it means

A row represents a state and a column represents an action, although a dictionary or another equivalent storage structure can serve the same purpose. Each stored Q-value estimates the return associated with taking that action and then following an optimal continuation under the task assumptions.
Tabular refers to the representation of the value function, not merely to using a spreadsheet or structured business data. The method does not automatically generalize an update from one state to another. A similar but separately indexed state retains its own estimate until experience or another explicit mechanism changes it.

How it works

Initialize the table, choose an action using a behavior rule, and observe reward R and next state S'. For a nonterminal transition, update the selected entry toward R + gamma times the maximum permitted next-state Q-value. With learning rate alpha, the new value is the old value plus alpha times the difference between that target and the old value.
For a true terminal transition, the target is just R. The next state's future value is zero because the task has ended. Q-learning is off-policy: its maximum-value target can differ from the exploratory action the behavior rule actually chooses next.
The table's size grows with the number of represented state-action combinations. Discretizing continuous measurements can make a table possible, but coarse bins may combine situations that require different actions, while fine bins demand more memory and experience.
Classical convergence guarantees require suitable finite-problem assumptions, adequate repeated exploration, and appropriate learning-rate conditions. A small table does not remove those requirements or guarantee that a poorly specified reward produces useful behavior.

Three practical examples

Teaching: trace every value update in a small grid-world or game.
Discrete simulation: learn decisions for a manageable set of resource and task states.
Algorithm checks: use a transparent tabular problem as a reference before introducing function approximation.

Think of it this way

Think of a notebook with a separate rating for every situation and available choice. Experience revises the relevant rating. A later rating can influence an earlier one when the earlier transition is revisited, but changing one page does not automatically rewrite every related page.

Worked example

Consider a fictional task with states S and T and terminal state G. At S, Advance gives reward 0 and moves to T, while Quit gives reward 2 and ends the task. At T, Finish gives reward 6 and ends the task. Initialize all three nonterminal state-action entries to 0. Use learning rate 0.50 and discount factor 0.80.
On the first episode, the behavior chooses Advance at S. Since Q(T,Finish) is still 0, its target is 0 + 0.80 times 0 = 0, and Q(S,Advance) remains 0. The behavior then chooses Finish at T and receives terminal reward 6. Its update gives Q(T,Finish) = 0 + 0.50 times (6 - 0) = 3.
That later update does not automatically change Q(S,Advance). On a second episode with the same action sequence, Advance now targets 0.80 times 3 = 2.4, so its entry becomes 1.2. Finish then updates from 3 toward 6, becoming 4.5.
On the third such episode, Advance targets 0.80 times 4.5 = 3.6 and updates from 1.2 to 2.4. Finish updates from 4.5 to 5.25. These calculations show reward information propagating backward through repeated sampled transitions, rather than appearing instantly in every table entry.
The true optimal values in this deterministic example are Q(T,Finish) = 6, Q(S,Advance) = 0.80 times 6 = 4.8, and Q(S,Quit) = 2. The finite updates have not yet reached those values. Moreover, Quit remains at its initial estimate until it is sampled; knowing its true value in this explanation does not mean the learner knows it.
For a separate storage illustration, 500 represented states with 8 actions require 4000 values. At 8 bytes per value, the raw numeric array uses 32000 bytes, excluding indexing and software overhead. Multiplying the number of states by 100 multiplies this storage and the number of distinct estimates by 100. This highlights why representation size and experience coverage matter together.

Illustrative case study

Fictional case: AlderRoutes uses a table whose state consists only of the robot's current room. The same room sometimes leads to different best actions depending on whether the robot carries a parcel, but that information is missing from the state index.
The team adds the relevant task condition and reevaluates the resulting larger state space. It also checks exploration coverage rather than assuming the extra entries will learn useful values immediately. This invented case illustrates state representation and makes no claim about an actual robot deployment.

Common misconceptions and mistakes

Common mistakes include updating every entry from one transition, treating unvisited zero values as established facts, and bootstrapping beyond true termination. Another error is assuming a discretized observation is automatically a valid Markov state. If relevant history is omitted, identical table indices can represent different decision situations, undermining the standard model assumptions.

Business use

Businesses can use tabular Q-learning for small, clearly defined sequential experiments and as an interpretable baseline. Before investing in a larger implementation, estimate the state-action count, data needed for coverage, and sensitivity to reward and state definitions. A transparent table is valuable for diagnosis, but it can become impractical when the decision space grows combinatorially.

Check your understanding

What is Q(T,Finish) after its first update?

Answer: Three.

Why does Q(S,Advance) stay zero during the first transition?

Answer: Its next-state estimate is still zero at that time.

What is the optimal Advance value in the stated task?

Answer: 4.8.

Related terms

Q-learning
Temporal-difference error
Exploration
Discretization
Markov decision process
Deep Q-Network
Q-table
action-value function
off-policy learning
state representation

Learn next

Next, implement the three-entry example and log update order explicitly. Then add an exploratory action-selection rule and inspect how unvisited or rarely visited entries affect the resulting policy.

Sources

Google for Developers, Machine Learning Glossary, tabular Q-learning: https://developers.google.com/machine-learning/glossary#tabular-q-learning Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, second edition, Section 6.5: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf

Previous: Tabular data Next: Tag (metadata)