Does the method require a model of transition probabilities?

Basic Monte Carlo evaluation uses sampled returns without an explicit transition model.

Can it update a running average?

Yes, after a usable return is observed; a full history of all returns is not required for a tabular average.

Is it the same as temporal-difference learning?

No. Temporal-difference methods bootstrap from estimated successor values, whereas basic direct estimation uses completed returns.

AI Glossary / Statistics and probability / Intermediate

Direct utility estimation

Definition

Direct utility estimation is a reinforcement-learning approach that estimates a state's value by averaging the observed returns following visits to that state. In the classic passive-learning setting, an agent follows a fixed policy and learns how much cumulative reward that policy produces from different states. It is a Monte Carlo style of policy evaluation: completed experience supplies the target values directly, without requiring a learned transition model or bootstrapping from another state's estimated value.

What it means

Here utility means expected cumulative reward under a specified policy and reward convention. It is not necessarily money, satisfaction or a general measure of human welfare. An immediate reward describes one step; a return includes subsequent rewards, often discounted. Following the same policy repeatedly can lead to different outcomes because the environment or policy is stochastic. Averaging those returns estimates the value of the policy, rather than automatically discovering the best possible policy or identifying the best action in every state.

How it works

Specify the policy, state representation, rewards, discount factor and episode termination rule. Generate or observe episodes that follow that policy, then calculate each relevant visit's remaining discounted return. Maintain a count and running average for each state, or fit a function to state-return pairs when using approximation. First-visit methods use the first occurrence of a state within an episode; every-visit methods use all occurrences. State which convention applies because repeated visits are not independent within an episode. Complete-return methods usually wait for termination. Finite truncation changes the target unless the omitted tail is appropriately addressed. Adequate visitation and a stable evaluation setting are essential.

Three practical examples

Simulated logistics: estimate expected discounted reward from different starting situations while following a fixed routing policy.
Game research: average completed game returns after visits to particular positions under a specified playing policy.
Agent evaluation: construct a transparent baseline for estimating the long-run value of a fixed decision procedure in a controlled episodic simulator.

Think of it this way

Think of estimating a route's value by repeatedly traveling it and recording each complete journey's outcome. You average what actually happened after the starting point instead of predicting each remaining segment from a detailed map. The result describes that route under those conditions, not every route you might have chosen.

Worked example

Suppose a hypothetical state is visited once in each of three completed episodes under the same policy. Using a convention where the first listed reward has discount exponent zero, the subsequent reward sequences are [1, 2, 4], [0, 3, 2] and [2, 1, 3]. With discount factor 0.9, their returns are 1 + 0.9 x 2 + 0.81 x 4 = 6.04; 0 + 0.9 x 3 + 0.81 x 2 = 4.32; and 2 + 0.9 x 1 + 0.81 x 3 = 5.33. Their average is 15.69/3 = 5.23. A fourth observed return of seven updates the average to 5.23 + (7 - 5.23)/4 = 5.6725. These few observations demonstrate the calculation, not precision, convergence or optimality of the policy.

Illustrative case study

Fictional case: Acorn Dispatch studies a fixed scheduling policy in a warehouse simulator. Analysts record completed episodes, calculate discounted returns and compare estimates across starting workload states. They discover that rarely visited congestion states have very few return samples, so they flag those estimates rather than treating all values as equally reliable. The team documents simulator assumptions and evaluates the reward definition before considering operational conclusions. This invented case illustrates policy evaluation and coverage checks, not a validated improvement in real warehouse performance.

Common misconceptions and mistakes

Do not confuse expected return with immediate reward. Direct utility estimation does not need explicit transition probabilities, but it still depends on the experience distribution and assumptions behind the state representation. Averaging data from changing policies does not automatically estimate the current policy's value. Off-policy evaluation needs additional treatment when behavior differs from the target policy. Long or highly variable episodes can make return estimates noisy, and unseen states receive no direct evidence. Learning state values alone does not solve exploration or policy improvement. The word direct does not mean exact, unbiased under every implementation or free of statistical uncertainty.

Business use

For businesses exploring sequential decision systems, the method provides an understandable evaluation baseline: outcomes are traced from observed rewards to returns to averages. That transparency can help detect incorrect discounting or termination handling. However, simulator quality, reward alignment and coverage of unusual states remain central. Compare methods on the relevant data and decision objective rather than assuming a simpler estimator is always better. Decisions involving real users should account for outcomes omitted from the reward and for differences between simulated and operational environments.

Check your understanding

Why is the estimated value 5.23 rather than the mean of the first rewards, which is one?

Answer: The value target includes discounted future rewards. Each episode contributes its complete return under the stated convention, so averaging only initial rewards would estimate a different quantity.

Related terms

Reinforcement learning
Return
Reward
Discount factor
State-value function
Policy
Episode
Monte Carlo policy evaluation
temporal-difference learning
first-visit method
every-visit method

Learn next

calculate discounted returns backward through a completed episode and verify reward indexing. Compare first-visit and every-visit estimates on an episode that revisits a state. Then contrast Monte Carlo evaluation with temporal-difference learning and investigate why evaluating a policy is distinct from improving it.

Sources

AIMA supporting notebook, Reinforcement Learning, direct utility estimation: https://zoo.cs.yale.edu/classes/cs370/aima/rltest.html Sutton and Barto, Reinforcement Learning: An Introduction, Monte Carlo methods: https://www.incompleteideas.net/book/bookdraft2018mar21.pdf Andrew Barto, Chapter 5: Monte Carlo Methods: https://people.cs.umass.edu/~barto/courses/cs687/Chapter%205.pdf

Previous: Direct sampling Next: Direct voice input