Can a state value be negative?

Yes. Expected future penalties can outweigh positive rewards.

Is the optimal state-value function the same as vπ?

The optimal value describes the best achievable expected return in the specified task; vπ evaluates one particular policy.

Must values be stored in a table?

No. A function approximator can estimate values for states represented by features.

AI Glossary / Mathematics and optimization / Intermediate

State-value function

Definition

A state-value function assigns each state the expected future return obtained by starting there and following a specified policy. In reinforcement learning, it summarizes the long-term consequences of being in that state under the chosen behavior.

What it means

A state describes the situation relevant to future decisions. A policy specifies how actions are selected. The state-value function, usually written vπ(s), therefore depends on both the environment and the policy π.

A location near a goal can have low value under a policy that repeatedly moves away from it. The same location can have high value under a better policy. State value is not an intrinsic score that remains unchanged when behavior changes.

Return combines future rewards according to a defined horizon and discount rule. It may include costs as negative rewards. Expected return averages over random outcomes and, when applicable, random action choices. A high expectation does not imply that every individual run ends well.

How it works

For a discounted task, define return from time t as Gt = R(t+1) + γR(t+2) + γ²R(t+3) + …, where γ is the discount factor. Then vπ(s) is the conditional expectation of Gt given that the current state is s and future actions follow π.

The Bellman expectation relationship decomposes that quantity into expected immediate reward plus discounted next-state value. In a finite Markov decision process, it averages over the policy's actions and the environment's possible transitions.

If the transition model is known, values can be computed through policy evaluation. If it is unknown, observed returns or temporal-difference updates can be used to estimate them. Estimated values carry sampling and approximation error.

Finite episodes need a terminal convention. Continuing discounted tasks commonly use γ below one with suitable reward bounds. An undiscounted infinite stream does not automatically have a finite value.

Three practical examples

A simulated warehouse robot evaluates the expected remaining reward from each location under its current routing policy.
A game-playing agent estimates whether a board position is favorable given its continuation strategy.
A scheduling experiment scores queue states by expected future reward, including waiting penalties, under a specified dispatch policy.

Think of it this way

Think of estimating the remaining journey quality from a junction while committing to a particular route-selection rule. The junction's prospects depend on what that rule does next and on uncertain travel conditions.

The analogy captures policy dependence and future consequences. A value function can combine abstract rewards rather than travel time, and an expected value is not a promise about a particular trip.

Worked example

A policy changes a state's value

Consider states S, W and a terminal state T. At W, the only action gives reward 6 and moves to T. At S, action A gives immediate reward 1 and moves to W. Action B gives immediate reward 4 and moves directly to T. Set γ = 0.5 and terminal value v(T) = 0.

The value at W is 6. Starting at S and choosing A yields return 1 + 0.5 × 6 = 4. Choosing B yields return 4. Both actions therefore have the same value in this setup.

Now use γ = 0.9. Action A has value 1 + 0.9 × 6 = 6.4, while action B remains worth 4. A policy that chooses A with probability 0.75 and B with probability 0.25 has vπ(S) = 0.75 × 6.4 + 0.25 × 4 = 5.8.

A policy that always chooses A has value 6.4 at S; one that always chooses B has value 4. The mixed policy's 5.8 is an expectation across its random action choice, not a return that either individual route produces. Discounting and policy must both accompany the reported number.

Illustrative case study

A fictional warehouse simulation labels locations with predicted state values. A reviewer notices that the dashboard compares values from different policy versions without identifying which policy produced each number.

The team adds policy identifiers, reward definitions and discount settings to the records. It also evaluates actual simulated returns across repeated runs rather than judging success from the learned value estimates alone.

When a new policy raises the average return but increases variability, the team reports that distinction. The lesson is that a value estimate is a conditional summary of a defined decision process. It is useful for planning, but it does not replace validation of the policy or assessment of undesirable outcomes.

Common misconceptions and mistakes

State value is the immediate reward. It includes future rewards under the return definition.
A state has one value regardless of policy. Different policies generally produce different values.
High expected value guarantees a good individual outcome. Random outcomes can differ substantially.
State value and action value are identical. Action value specifies a first action as well as the starting state.
A learned value is automatically correct. Function approximation and limited experience can introduce error.

Business use

State-value functions help explain how reinforcement-learning systems evaluate future consequences. For a business simulation, define the state, reward, horizon and policy before interpreting a value number. Ensure the state representation retains information needed for the modeling assumptions.

Review values alongside observed returns, uncertainty and decision-specific constraints. Comparing values across systems is meaningful only when their reward scales and return conventions align. A larger number under a changed reward definition is not by itself evidence that the resulting behavior serves the original objective better.

Check your understanding

Why is vπ(S) = 5.8 in the mixed-policy example even though neither route returns 5.8?

Answer: It is the probability-weighted average of returns 6.4 and 4. An expectation need not equal any single realized outcome.

Related terms

Learn next

Read action-value function and the Bellman equation. Recalculate the example when action B pays 5, then determine which policy is preferred at each discount factor. State clearly whether you are evaluating a fixed policy or searching for a better one.

Sources

Aalto Dictionary of Machine Learning, state-value function, policy evaluation and Bellman operator: https://aaltodictionaryofml.github.io/ADictML_Main.pdf. Source definitions reviewed. The three-state calculation and fictional warehouse review are original examples; the supplementary Sutton–Barto textbook was not accessible through the available web retrieval.

Previous: State-space search Next: Statement