Why does SARSA need the next action?

Its target uses that selected action's value, which makes the basic update on-policy.

Can SARSA use a neural value model?

Yes, though the learning behavior and guarantees depend on the approximation and training design.

Is the reward the same as the action value?

No. The reward is one observed signal; the action value estimates discounted cumulative reward over future interaction.

AI Glossary / Graphs and symbolic AI / Intermediate

SARSA

Definition

SARSA is an on-policy temporal-difference reinforcement-learning algorithm that updates an action-value estimate using the next action actually selected by the current policy. Its name abbreviates State, Action, Reward, next State and next Action.

What it means

An agent learns through interaction: it observes a state, chooses an action, receives a reward and reaches another state. SARSA uses that transition plus the next chosen action to revise its estimate of the earlier action's long-term value.

On-policy means that the update evaluates the behavior policy being followed. If the policy sometimes explores, the consequences of those exploratory actions enter the learning target. This differs from the standard one-step Q-learning target, which uses the highest estimated next-action value regardless of which action was actually selected.

SARSA is often introduced with a table containing one value per state-action pair. Function approximation is also possible, but properties established for a tabular setting should not automatically be transferred to arbitrary neural implementations.

How it works

Choose an action A in state S using the current policy, such as an epsilon-greedy policy. Execute it, observe reward R and next state S', then choose the next action A' using that same policy.

The one-step update is Q(S,A) ← Q(S,A) + alpha × [R + gamma Q(S',A') − Q(S,A)]. Alpha controls the update size and gamma discounts future reward. The bracketed quantity is the temporal-difference error.

At a true terminal state, there is no continuing action value, so the target is just the terminal transition's reward. An artificial time-limit truncation may require different handling if the underlying task continues.

The agent repeats this process while its estimates and policy evolve. Convergence claims require conditions on the task, exploration, step sizes and representation. Continued fixed exploration evaluates a policy with exploration; it should not be casually described as identical to a fully greedy deployed policy.

Three practical examples

1. Grid-world learning: A simulated agent learns routes while occasionally exploring. SARSA's values reflect the consequences of the actions that exploration actually selects.

2. Educational control: A small simulator with discrete actions allows learners to compare on-policy and off-policy updates using the same observed transition.

3. Resource-allocation research: A simplified environment supplies rewards for allocations over time. A SARSA baseline helps investigate how exploratory behavior changes learned action values under explicit assumptions.

Think of it this way

Think of learning the value of a route based on the turns you actually tend to take, including occasional exploratory detours. The estimate reflects that behavior rather than assuming every later turn will be the currently best one.

The analogy does not imply that SARSA remembers whole routes or guarantees cautious behavior. It updates numerical values from experience, and its result depends on the policy, rewards and environment.

Worked example

Suppose Q(S,A) is 4, the observed reward is 2, gamma is 0.9 and alpha is 0.5. In the next state, the policy actually selects action B with estimated value Q(S',B) = 3. Another available action C has estimated value 7.

SARSA uses the selected action B. Its target is 2 + 0.9 × 3 = 4.7. The error is 4.7 − 4 = 0.7, so the updated value is 4 + 0.5 × 0.7 = 4.35.

A standard Q-learning update on the same transition would instead use the maximum next-action value 7. Its target would be 8.3 and updated value 6.15.

Neither calculation proves which method will perform better on the task. The difference shows what each update assumes about subsequent behavior. If S' were terminal, both targets would instead be 2 under the stated episodic convention.

Illustrative case study

In a fictional project, Alder Learning compares two navigation agents in a simulator. The team initially reports each agent's training reward without recording how often it explores or how evaluation actions are selected.

After reviewing the experiment, it separates behavior during learning from the policy used for evaluation. It reports the exploration schedule, reward design and terminal-state handling, then compares results over multiple runs.

The case illustrates that an on-policy learner's training behavior is part of the experiment. Removing exploration at evaluation time changes the policy and should be described explicitly rather than treated as an irrelevant implementation detail.

Common misconceptions and mistakes

SARSA uses the maximum value in the next state. The basic update uses the value of the next selected action.
On-policy means no exploration. An exploratory policy can be the policy being learned about.
One update produces the true action value. Estimates require repeated experience and suitable conditions.
SARSA is inherently safe. Reward design and exploration can still produce undesirable actions.
Expected SARSA is identical to sampled SARSA. Expected SARSA averages over next actions according to the policy instead of using just the sampled one.

Business use

SARSA is useful as a clear baseline for understanding sequential learning and the effect of behavior during exploration. It is most informative when the environment, reward and action space are explicitly defined.

Before applying it to an operational problem, check whether simulation adequately represents important consequences and whether experimentation is acceptable. A small successful tabular example does not establish that a large partially observed workflow satisfies the same assumptions.

For comparisons, record policy selection, exploration rates, learning rates and evaluation conditions. Inspect learning curves across repeated runs rather than treating one favorable episode as evidence of convergence. Separate the algorithm's update rule from the broader controls that determine which actions the agent is allowed to take.

Check your understanding

Why does SARSA use 3 rather than 7 in the worked target?

Answer: The policy selected action B, whose estimated value is 3. The basic SARSA target follows that selected action rather than substituting the highest available value.

Related terms

Q-learning
Epsilon greedy policy
Expected SARSA
Discount factor
Terminal states
Temporal-difference learning
on-policy learning
action-value function
terminal state

Learn next

Repeat the worked update with the policy selecting C instead of B. Then compare a terminal transition with a continuing transition. Explain which changes affect the target and which affect only the size of the update.

Sources

MathWorks documents SARSA's selected-action target and tabular update: https://www.mathworks.com/help/reinforcement-learning/ug/sarsa-agents.html

De Asis and colleagues place SARSA among related temporal-difference control algorithms: https://arxiv.org/abs/1703.01327

The numerical update and Alder Learning case are original illustrations, not reported experiments.

Previous: Sargan - Hansen test Next: SARSA(lambda)