Must rewards be positive?

No. Negative rewards can represent costs, penalties, or undesirable outcomes.

Is the next reward always determined by the current state?

Not necessarily. It may depend on the transition or be random; r(s) can summarize its conditional expectation for value evaluation.

Does a high-value state mean the process always performs well from there?

No. It describes expected return. Individual trajectories can vary substantially around that expectation.

AI Glossary / Graphs and symbolic AI / Intermediate

Markov reward process

Definition

A Markov reward process, or MRP, is a Markov state-transition model equipped with rewards. It describes how a system moves between states and the rewards accumulated along the way, without an explicit choice of actions. Its value function measures the expected future return from each state under the specified process.

What it means

Suppose a support ticket can be Open, UnderReview, or Closed. A transition model describes how tickets move between these states. Attaching a handling cost to each stage turns the model into a reward process, with costs represented as negative rewards if desired.

The current state must contain the information needed for the model's next-step probabilities. If ticket age changes the chance of closure but is absent from the state, a simple status-only model may fail this assumption. The phrase “Markov” does not make the assumption automatically true; it identifies a modeling requirement to examine.

How it works

A common finite MRP specifies states, transition probabilities, expected immediate rewards, and a discount factor γ. Its return adds future rewards with successive powers of γ. State value is the expected return conditional on starting in that state.

The Bellman expectation equation is v(s)=r(s)+γΣP(s,s′)v(s′). It combines the expected next reward with the discounted expected value of the successor state. For a finite discounted model with bounded rewards and 0≤γ<1, the values can be found by solving the corresponding linear equations or using iterative evaluation.

An MRP contains no action maximization. Fixing a stationary policy in a Markov decision process induces a reward process by averaging transitions and rewards over that policy's action choices. This evaluates specified behavior rather than searching for better behavior.

Three practical examples

Service operations: estimate accumulated handling costs for cases moving through a fixed support workflow.
Equipment monitoring: calculate expected future operating benefits and downtime penalties under an established maintenance policy.
Game analysis: evaluate expected future scores when a player follows a fixed strategy, before considering whether to improve that strategy.

Think of it this way

Think of riding a train through stations where each stop adds a credit or charge to your journey. The route changes randomly according to a timetable of probabilities, and you evaluate what the journey is worth from each station.

The analogy stops at decision-making: if you choose which train to take at every station, the model needs actions and becomes a decision problem. A reward process describes the route behavior already specified.

Worked example

Consider states Open O, Review R, and Closed C. From O, a case stays O with probability 0.5 and moves to R with probability 0.5. From R it always moves to C; C remains C. Each transition out of O has reward −2, each transition out of R has reward −3, and C has reward 0. Use γ=0.9.

The closed-state value is v(C)=0. The review value is v(R)=−3+0.9×0=−3. For Open, the equation is v(O)=−2+0.9[0.5v(O)+0.5(−3)]. Therefore v(O)=−3.35+0.45v(O), giving 0.55v(O)=−3.35 and v(O)≈−6.091.

A particular path O→R→C earns −2+0.9(−3)=−4.7. A longer path O→O→R→C earns −2+0.9(−2)+0.9²(−3)=−6.23. Neither path alone equals the expected value across all possible paths. Reward timing is explicit here: the immediate reward depends on the state being left. Another convention is possible, but its equations must match it.

Illustrative case study

Fictional case: a small repair service models equipment as Operating, WaitingForParts, or Returned. It uses an MRP to evaluate the existing repair workflow's accumulated delay penalties. The first model assigns the same transition probabilities to every waiting job.

Reviewers find that jobs waiting for common parts behave differently from those needing rare imports. The team expands the state representation and re-estimates the transitions before using the values for planning. It also separates descriptive evaluation from choosing a new ordering policy. The fictional case illustrates why better state definitions can matter more than obtaining a precise solution to an oversimplified model.

Common misconceptions and mistakes

Do not confuse reward with value. Reward concerns one step; value includes the expected stream of future rewards. Do not confuse a realized return with its expectation, or insert a maximum over actions into a model that has none.

Discounting is a modeling choice with consequences for distant outcomes. It is not a substitute for representing uncertain transitions. An undiscounted continuing process can have infinite total return; using γ=1 requires suitable conditions or a different objective such as average reward. Absorbing termination with zero subsequent rewards should be represented consistently.

Business use

MRPs provide a useful baseline for understanding the long-run consequences of an existing process. A team can compare how changes in estimated delays or per-stage costs affect expected return, while keeping the behavior being evaluated explicit.

Before interpreting the resulting numbers, review reward units, time-step duration, state sufficiency, and stability of transition estimates. If a manager wants to choose actions dynamically, extend the analysis to an MDP rather than treating the current MRP's values as an optimized recommendation. Model uncertainty also remains even when the linear equations are solved exactly.

Check your understanding

in the example, why is v(Open) more negative than its immediate reward of −2?

Answer: Open can be revisited, generating repeated costs, and the process eventually incurs the Review cost. Its value includes the discounted expectation of those later rewards as well as the next one.

Related terms

Learn next

set the example's discount factor to zero and verify that each value becomes its immediate expected reward. Then add two alternative actions at Open and consider why evaluating each fixed policy differs from finding the best policy. This leads directly to Markov decision processes.

Sources

David Silver, “Lecture 2: Markov Decision Processes,” MRP definitions and Bellman expectation equations, primary lecture slides read via the Berkeley-hosted copy: https://ecal.studentorg.berkeley.edu/tbsi/References/DSilver-UCL-RL/MDP.pdf. The author's own PDF URL did not retrieve successfully. The workflow, arithmetic, and fictional case are original teaching examples.

Previous: Markov random field Next: Martingale (probability theory)