Does it need the sampled next action for its target?

No; it uses the policy-weighted next-action values.

Does it remove all randomness?

No; it removes a specific conditional source of sampling variability.

When does it match Q-learning's target?

When the target policy puts its probability on maximizing actions.

AI Glossary / Reinforcement learning and robotics / Advanced

Expected SARSA

Definition

Expected SARSA is a temporal-difference method that updates an action-value estimate using the policy-weighted average of the next state's action values. Ordinary SARSA uses the value of one sampled next action; Expected SARSA averages over the possible next actions under a specified policy. This removes the sampling variability associated with that next-action choice, conditional on the observed transition and current estimates.

What it means

Suppose a policy sometimes explores a less valuable action. Its expected next-state value reflects that possibility rather than assuming the best action will always be chosen. Expected SARSA incorporates those action probabilities directly into its learning target.
The expectation is over next actions, not automatically over every possible next state or reward. The algorithm can still learn from a sampled environmental transition, and its target can remain noisy because the environment is stochastic.

How it works

For an observed nonterminal transition, compute y = reward + gamma sum_a pi(a|next_state) Q(next_state,a). Then update Q(current_state,current_action) <- Q(current_state,current_action) + alpha [y - Q(current_state,current_action)]. At an actual terminal state, use the immediate reward without bootstrapping.
The probabilities in the sum must describe the policy being evaluated or improved. If behavior follows that same policy, the method is on-policy. It can also use a different behavior policy, with suitable coverage; the expectation then refers to the target policy. If that target always chooses a maximizing action, the expected backup reduces to the Q-learning maximum.
Computing the sum requires access to the next-action probabilities and values. For a small discrete action set this is straightforward. Large or continuous action spaces can make exact expectation more expensive or require another computational approach.

Three practical examples

Simulated navigation: a learner accounts for the chance that its exploratory policy selects a less favorable next move.
Discrete scheduling: a policy mixes several action choices, and its update uses their probability-weighted values rather than one sampled choice.
Algorithm comparison: researchers compare SARSA and Expected SARSA on the same transitions to isolate variability from next-action sampling while keeping the policy fixed.

Think of it this way

Imagine estimating the outcome of a choice that is followed by a lottery over several next choices. Instead of drawing one ticket and using only its estimated value, you average the tickets' values using their probabilities. You still need reliable values and probabilities, and the earlier part of the situation may remain uncertain.

Worked example

Consider an original nonterminal transition with reward 1, discount gamma = 0.9 and current state-action estimate 4. At the observed next state, the action values for A, B and C are 8, 5 and -1. The target policy assigns probabilities 0.6, 0.3 and 0.1. Set alpha = 0.2.
The expected next-action value is 0.6 times 8 + 0.3 times 5 + 0.1 times (-1) = 4.8 + 1.5 - 0.1 = 6.2. The Expected SARSA target is therefore 1 + 0.9 times 6.2 = 6.58. The updated estimate is 4 + 0.2 times (6.58 - 4) = 4.516.
Ordinary SARSA would use whichever next action was sampled. If it sampled A, its target would be 8.2 and the updated estimate would be 4.84. If it sampled B, the target would be 5.5 and the update would yield 4.3. If it sampled C, the target would be 0.1 and the update would yield 3.22.
Averaging those three possible updated estimates under the same policy gives 0.6 times 4.84 + 0.3 times 4.3 + 0.1 times 3.22 = 4.516, matching the Expected SARSA update. The match concerns this fixed transition, policy and value table, not an assertion that complete training runs will follow identical paths.
The sampled next-action value has second moment 0.6 times 64 + 0.3 times 25 + 0.1 times 1 = 46. Its variance is 46 - 6.2 squared = 7.56. Multiplying by gamma squared gives next-action-induced target variance 0.81 times 7.56 = 6.1236. Expected SARSA removes that particular conditional variance by using the exact average.
A greedy target would instead use 8, giving the Q-learning target 8.2. It differs because it represents a different next-action policy. If the transition were terminal, all these future-action calculations would be omitted: the target would be 1 and the updated estimate would be 3.4.
These are exact arithmetic comparisons using stipulated estimates. They do not show that the supplied Q values are correct or that Expected SARSA will outperform another method on a particular task.

Illustrative case study

Fictional company Cedar Scheduling studies a controller whose policy deliberately randomizes among three dispatch rules. Its analysts record the next-state action probabilities and verify that the probabilities used in the backup match the intended target policy.
They compare update variability under fixed data and separately evaluate the resulting policies. They also record computation time, since evaluating all actions has a cost. A smoother sequence of updates is treated as a diagnostic observation rather than automatic proof of better operational decisions. This fictional workflow reports no measured performance gain.

Common misconceptions and mistakes

Expected SARSA does not mean taking an unweighted average unless the policy is uniform. It also does not mean averaging over unknown environmental dynamics merely because the word expected appears in its name.
Confusing the behavior policy with the target policy can silently change the learning objective. Another mistake is assuming that removing next-action sampling removes all variance or approximation error. Rewards, successor states and estimated values can still be uncertain. Finally, the method need not always be on-policy, and a greedy target connects it directly to Q-learning.

Business use

Expected SARSA is useful for explaining how learning can account for a deliberately randomized decision policy. Evaluation should identify the target probabilities, exploration convention, action set and terminal handling. Comparisons should measure prediction or decision quality as well as update variability and computation. This helps distinguish the benefit of averaging from changes caused by evaluating a different policy.

Check your understanding

Why was the expected next value 6.2?

Answer: The values were weighted by 0.6, 0.3 and 0.1.

Why did the SARSA updates average to 4.516?

Answer: They used the same policy, transition and starting estimate.

Why did the terminal update omit the average?

Answer: There was no future action value to bootstrap.

Related terms

SARSA
Q-learning
Exploration
Temporal-difference learning
Target policy
Behavior policy
Expected backup

Learn next

Build an epsilon-greedy action distribution, including its tie-breaking convention, and compute the corresponding expected backup.

Sources

Sutton and Barto, Reinforcement Learning: An Introduction, second edition, Section 6.6 and Equation 6.9: https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf . The action-value table and company scenario are original illustrations.

Previous: Expected monetary value Next: Expected utility