Is sigma the same as lambda?

No. Sigma mixes sampling and expectation; lambda typically mixes backup horizons through traces or returns.

Does Q(sigma) need an environment model?

The described sample-based method does not require transition probabilities.

Can sigma change within an episode?

Yes, provided the selected algorithm consistently uses those step-specific values.

AI Glossary / Reinforcement learning and robotics / Advanced

N-step Q(sigma)

Definition

N-step Q(sigma) is a family of multi-step action-value learning algorithms that mixes sampled action backups with expectations over actions. The parameter sigma controls the mixture at each step. With sigma equal to one throughout, it recovers n-step Sarsa; with sigma equal to zero throughout, it recovers n-step tree backup. Intermediate choices connect the two approaches.

What it means

After taking an action, a learner can use the value of the particular next action it sampled, or average the values of available actions under the target policy. Q(sigma) makes that choice adjustable instead of fixing it for every backup.

The n parameter controls how far experience is propagated through a backup. Sigma controls how action sampling and expectation are combined within that backup. Neither parameter is the discount factor gamma, and sigma is not the exploration probability. A learner can hold its behavior policy fixed while changing sigma to study the learning update itself.

How it works

At a nonterminal next state, calculate its expected action value V by summing policy probabilities times action values. A mixed one-step TD error uses reward plus gamma times the combination sigma Q(next sampled action) plus (1 minus sigma) V, minus the current action value.

For a multi-step return, combine these errors along the observed trajectory. In the fixed-estimate on-policy form, each additional link multiplies later errors by gamma times [sigma plus (1 minus sigma) times the target probability of the action actually taken]. Terminal transitions omit bootstrap values.

Sigma may be constant or vary by step. All-one sigma uses the sampled reward sequence as in Sarsa. All-zero sigma incorporates expected values at every branch, producing tree backup. Sampling intermediate steps but taking expectation only at the final bootstrap gives another familiar multi-step Expected Sarsa construction.

Off-policy versions require the corresponding correction machinery and coverage assumptions. Simply substituting another target policy into a sampled-action mixture is insufficient. The illustration below deliberately uses an on-policy setting and fixed action-value estimates, avoiding complications from changing estimates during a delayed update.

Three practical examples

1. A tabular navigation experiment compares full sampling, full expectation and intermediate mixtures on the same stochastic environment.

2. A simulator team studies whether a longer backup spreads delayed rewards effectively without introducing excessive variability.

3. An algorithm researcher varies sigma during training and checks whether the change improves performance across repeated runs rather than one favorable trajectory.

Think of it this way

Think of combining a traveler's actual route report with estimates for the roads they could have taken. Sarsa emphasizes the reported sequence. Tree backup blends in alternative-road estimates at each junction. Q(sigma) adjusts that balance while preserving the accounting needed to carry later information back through earlier junctions.

Worked example

Consider an original two-step on-policy example with gamma equal to 0.9 and estimates held fixed. At the initial state-action pair, Q is 2. Taking it gives reward 1 and reaches S1.

At S1, action A has policy probability 0.25 and Q-value 4. Action B has probability 0.75 and Q-value 8. The sampled action is A, and taking it produces reward 6 followed by termination. The expected action value at S1 is 0.25 times 4 plus 0.75 times 8, which is 7.

Set sigma to 0.5. The first mixed TD error is 1 plus 0.9 times [0.5 times 4 plus 0.5 times 7] minus 2, giving 3.95. The terminal TD error is 6 minus 4, giving 2.

The coefficient carrying that terminal error back is 0.9 times [0.5 plus 0.5 times 0.25], or 0.5625. The two-step target is therefore the initial Q of 2 plus 3.95 plus 0.5625 times 2, giving 7.075.

A backward calculation confirms it: the continuation contribution is 0.625 times the observed terminal return 6 plus 0.375 times the unchosen action estimate 8. This equals 6.75; adding reward 1 and applying gamma gives 1 plus 0.9 times 6.75, again 7.075.

At sigma one, the target is 1 plus 0.9 times 6, or 6.4. At sigma zero, it is 1 plus 0.9 times [0.25 times 6 plus 0.75 times 8], or 7.75. With learning rate 0.2, the mixed target updates the initial estimate to 2 plus 0.2 times (7.075 minus 2), or 3.015. These are update calculations, not claims about the true action values.

Illustrative case study

In a fictional project, Willow Simulation trains a controller for a small routing environment. The team notices that fully sampled multi-step targets fluctuate substantially, while expectation-heavy targets depend strongly on inaccurate early action estimates.

It compares several sigma schedules with the same exploration policy and reports variation across runs. A schedule is retained only if it improves separate evaluation episodes. The team records n, gamma, sigma and the behavior policy independently, so an apparent gain cannot be attributed vaguely to changing the amount of exploration when the actual change was in the backup rule.

Common misconceptions and mistakes

Sigma zero does not remove all randomness. Rewards and environment transitions are still sampled; the expectation concerns actions in the backup.

Do not identify tree backup with taking an expectation only at the last step. It incorporates policy-weighted branches throughout the return.

Intermediate sigma values are not universally superior. Their usefulness depends on estimate quality, sampling variability, backup length and the task. Also avoid extending a tabular convergence argument automatically to arbitrary neural approximators or uncontrolled off-policy data.

Business use

The method is most relevant to teams designing reinforcement-learning estimators and benchmarking how efficiently they learn from sequential experience. Business evaluation should examine both data use and actual task performance. A useful experiment isolates the backup mixture from changes in exploration, reward design and model size, making the source of any improvement easier to interpret and reproduce.

Check your understanding

Why is the terminal error multiplied by 0.5625?

Answer: It passes through one discounted mixed sampling-and-policy-probability link.

What changes at sigma one?

Answer: The backup becomes fully sampled n-step Sarsa in the stated on-policy setting.

Related terms

Expected SARSA
n-step Sarsa
tree backup
action-value function
multi-step return
importance sampling and temporal-difference learning

Learn next

Next, repeat the example with the other action sampled. Then derive a three-step backup to see how the mixed coefficients multiply along the trajectory.

Sources

Primary source: Kristopher De Asis and colleagues, Multi-Step Reinforcement Learning: A Unifying Algorithm, especially the Q(sigma) derivation: https://arxiv.org/pdf/1703.01327

The two-step calculation and fictional case are original teaching material. The worked example uses fixed estimates and on-policy sampling; it does not substitute for a complete off-policy implementation.

Previous: N-pairs loss Next: n-step return